US-12626102-B2 - Encoding and reconstructing inputs using neural networks

US12626102B2US 12626102 B2US12626102 B2US 12626102B2US-12626102-B2

Abstract

Systems, methods, devices, and other techniques are described herein for training and using neural networks to encode inputs and to process encoded inputs, e.g., to reconstruct inputs from the encoded inputs. A neural network system can include an encoder neural network, a trusted decoder neural network, and an adversary decoder neural network. The encoder neural network processes a primary neural network input and a key input to generate an encoded representation of the primary neural network input. The trusted decoder neural network processes the encoded representation and the key input to generate a first estimated reconstruction of the primary neural network input. The adversary decoder neural network processes the encoded representation without the key input to generate a second estimated reconstruction of the primary neural network input. The encoder and trusted decoder neural networks can be trained jointly, and these networks trained adversarially to the adversary decoder neural network.

Inventors

Martin Abadi
David Godbe Andersen

Assignees

GOOGLE LLC

Dates

Publication Date: 20260512
Application Date: 20231109

Claims (15)

1 . A computer-implemented method, comprising: receiving (i) an encoded representation of a first input and (ii) a second input that represents a private decoding key distinct from the encoded representation of the first input, wherein the encoded representation of the first input was generated by an asymmetric encoder neural network by processing (i) the first input and (ii) a third input that represents a public encoding key distinct from the first input and the second input; and processing, by an asymmetric decoder neural network, the encoded representation of the first input and the second input that represents the private decoding key to generate an estimated reconstruction of the first input.
2 . The computer-implemented method of claim 1 , wherein: the third input is derived from the second input according to a first pre-defined set of transformations, the second input is derived from the third input according to a second pre-defined set of transformations, or the second input and the third input are both derived using a common procedure so as to be complementary to each other.
3 . The computer-implemented method of claim 1 , wherein processing, by the asymmetric decoder neural network, the encoded representation of the first input and the second input to generate the estimated reconstruction of the first input comprises decoding the encoded representation of the first input using the second input.
4 . The computer-implemented method of claim 1 , wherein the asymmetric decoder neural network comprises one or more fully-connected layers followed by one or more convolutional layers.
5 . The computer-implemented method of claim 1 , wherein the encoded representation of the first input is an encoded representation of at least one of a text sequence or one or more numbers.
6 . The computer-implemented method of claim 1 , wherein: the asymmetric decoder neural network is further configured to process the encoded representation of the first input and the second input to generate an indication of whether the encoded representation of the first input has been tampered with subsequent to initial generation of the encoded representation of the first input by the asymmetric encoder neural network; and the method further comprises providing the indication of whether the encoded representation of the first input has been tampered with.
7 . The computer-implemented method of claim 1 , wherein the first input is derived from an initial input according to a pre-defined set of transformations.
8 . The computer-implemented method of claim 1 , wherein the estimated reconstruction of the first input is an estimation of one or more values derived from the first input according to a pre-defined set of transformations.
9 . The computer-implemented method of claim 1 , wherein the asymmetric decoder neural network is a first asymmetric decoder neural network that has been adversarially trained with respect to a second asymmetric decoder neural network, wherein the second asymmetric decoder neural network is trained to generate a second estimated reconstruction of the first input without processing the second input.
10 . A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving (i) an encoded representation of a first input and (ii) a second input that represents a private decoding key distinct from the encoded representation of the first input, wherein the encoded representation of the first input was generated by an asymmetric encoder neural network by processing (i) the first input and (ii) a third input that represents a public encoding key distinct from the first input and the second input; and processing, by an asymmetric decoder neural network, the encoded representation of the first input and the second input that represents the private decoding key to generate an estimated reconstruction of the first input.
11 . The system of claim 10 , wherein: the third input is derived from the second input according to a first pre-defined set of transformations, the second input is derived from the third input according to a second pre-defined set of transformations, or the second input and the third input are both derived using a common procedure so as to be complementary to each other.
12 . The system of claim 10 , wherein processing, by the asymmetric decoder neural network, the encoded representation of the first input and the second input to generate the estimated reconstruction of the first input comprises decoding the encoded representation of the first input using the second input.
13 . The system of claim 10 , wherein the asymmetric decoder neural network comprises one or more fully-connected layers followed by one or more convolutional layers.
14 . The system of claim 10 , wherein the encoded representation of the first input is an encoded representation of at least one of a text sequence or one or more numbers.
15 . One or more non-transitory computer-readable media storing instructions that are operable, when executed by one or more processors, to cause the one or more processors to perform operations comprising: receiving (i) an encoded representation of a first input and (ii) a second input that represents a private decoding key distinct from the encoded representation of the first input, wherein the encoded representation of the first input was generated by an asymmetric encoder neural network by processing (i) the first input and (ii) a third input that represents a public encoding key distinct from the first input and the second input; and processing, by an asymmetric decoder neural network, the encoded representation of the first input and the second input that represents the private decoding key to generate an estimated reconstruction of the first input.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is a continuation application of pending U.S. application Ser. No. 17/685,559, filed Mar. 3, 2022, which is a continuation application of U.S. application Ser. No. 16/323,205, filed Feb. 4, 2019, now U.S. Pat. No. 11,308,385, issued Apr. 19, 2022, which is a National Stage Application under 35 U.S.C. § 371 and claims the benefit of International Application No. PCT/US2017/045329, filed Aug. 3, 2017, which claims priority to U.S. Provisional Application No. 62/370,954, filed Aug. 4, 2016. The disclosure of the foregoing applications are incorporated by reference in their entireties. TECHNICAL FIELD This specification generally describes systems, methods, devices, and other techniques for training and using neural networks to encode inputs and to process encoded inputs, e.g., to reconstruct inputs from the encoded inputs. BACKGROUND Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. Neural networks have been trained to perform various data processing tasks, such as classification, prediction, and translation. Some systems include multiple data processing components, e.g., in successive stages, to carry out a given task. SUMMARY This specification discloses neural networks and neural network training systems that can be implemented as computer programs on one or more computers in one or more locations. For example, neural networks are described that apply a neural network input key to obfuscate a neural network input in order to obfuscate (e.g., encode) the neural network input and prevent it from being reconstructed by another neural network. For example, an encoder neural network and a trusted decoder neural network can be jointly trained to encode and decode, respectively, a primary neural network input using a shared secret key or a public-private key pair. The encoder neural network and the trusted decoder neural network can further be trained adversarially with respect to an adversary decoder neural network that attempts to decode the output of the encoder neural network, although the adversary decoder neural network may not have access to the decoding key. The training system may not only encourage the first decoder neural network to generate accurate reconstructions of the original neural network input, but may also reward the encoder neural network for generating encoded representations of the neural network input that cannot be accurately decoded by the adversary decoder neural network. In some implementations, a neural network training system trains an encoder neural network and a first decoder neural network. The encoder neural network is trained to generate an encoded representation of a first input by processing the first input and a second input that is distinct from the first input. The first decoder neural network is trained to generate an estimated reconstruction of the first input by processing the encoded representation of the first input and either the second input or a third input that is related to the second input. In some examples, the encoder neural network and first decoder network are configured to encrypt and decrypt the first input, respectively. The encoder neural network generates the encoded representation of the first input (e.g., encrypts the first input) using an encryption key and the first decoder neural network generates the estimated reconstruction of the first input (e.g., decrypts the encoded representation of the first input) using a corresponding decryption key. The encryption and decryption keys may be the same for symmetric encryption or complementary (e.g., a public-private key pair) for asymmetric encryption. The training system can train the encoder neural network and the first decoder neural network to protect the secrecy of all or a portion of the first input. For example, the encoder neural network and the first decoder neural network can be trained in conjunction with a second decoder neural network that represents an adversary. Generally, the second decoder neural network attempts to compromise the secrecy of all or a portion of the first input, e.g., by attempting to reconstruct the first input from the encoded representation of the first input. To disrupt the ability of the second decoder neural network to achieve its goal, the training system can, for example, adjust the parameters of the encoder neural network and the first decoder neural network using a loss function that encourages accurate reconstruction of inputs by the first decoder neural network, but discour