Skip to content

Commit ae59c7f

Browse files
harshachintagauravpurohit06gcf-owl-bot[bot]
authored
feat: add support for Proto columns (#1991)
* feat: Support For Proto Column in Spanner (#1829) * feat: Generated Proto Changes Changes need to be reverted. * feat: Implementation for Proto Message & Enum -Adding Logic for Serialization & Deserialization -New Type Codes and utilities * feat: Proto static files and typings and generated descriptors * sample: Adding DML, DQL, DML, table insert & read samples. * style: Lint * test: Adding unit tests * refactor: minor refactoring * refactor: minor refactoring * test: Adding integration tests * docs: Adding docs * test: Adding sample Integration Tests * refactor: Minor refactoring and updating comments/docs. * test: Making test fixes * refactor: Minor refactoring and lint fixes * refactor: Minor refactoring and lint fixes * Updating docs and minor changes. * test: fixing test * refactor: minor changes * refactor: minor refactoring * docs: Updating docs & comments * feat(spanner): fix lint * fix(spanner: lint * fix(spanner): lint * fix(spanner): lint * fix(spanner): lint * feat(spanner): fix db name * feat(spanner): remove it.only * feat(spanner): fix tests lint * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): remove samples and sample tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): update schema to avoid reserved keyword * feat(spanner): add samples and sample tests * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): code refactoring --------- Co-authored-by: Gaurav Purohit <purohitgaurav0612@gmail.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent 81fa610 commit ae59c7f

25 files changed

+2724
-16
lines changed

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,10 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-spanner/tre
167167
| Alters a sequence in a PostgreSQL database. | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/pg-sequence-alter.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/pg-sequence-alter.js,samples/README.md) |
168168
| Creates sequence in PostgreSQL database. | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/pg-sequence-create.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/pg-sequence-create.js,samples/README.md) |
169169
| Drops a sequence in PostgreSQL database. | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/pg-sequence-drop.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/pg-sequence-drop.js,samples/README.md) |
170+
| Proto-query-data | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-query-data.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-query-data.js,samples/README.md) |
171+
| Creates a new database with a proto column and enum | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-type-add-column.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-type-add-column.js,samples/README.md) |
172+
| Proto-update-data-dml | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-update-data-dml.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-update-data-dml.js,samples/README.md) |
173+
| Proto-update-data | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-update-data.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-update-data.js,samples/README.md) |
170174
| Queryoptions | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/queryoptions.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/queryoptions.js,samples/README.md) |
171175
| Quickstart | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/quickstart.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/quickstart.js,samples/README.md) |
172176
| Read data with database role | [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/read-data-with-database-role.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/read-data-with-database-role.js,samples/README.md) |

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@
3838
"ycsb": "node ./benchmark/ycsb.js run -P ./benchmark/workloada -p table=usertable -p cloudspanner.instance=ycsb-instance -p operationcount=100 -p cloudspanner.database=ycsb",
3939
"fix": "gts fix",
4040
"clean": "gts clean",
41-
"compile": "tsc -p . && cp -r protos build",
41+
"compile": "tsc -p . && cp -r protos build && cp -r test/data build/test",
4242
"prepare": "npm run compile-protos && npm run compile",
4343
"pretest": "npm run compile",
4444
"presystem-test": "npm run compile",

samples/README.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,10 @@ and automatic, synchronous replication for high availability.
9292
* [Alters a sequence in a PostgreSQL database.](#alters-a-sequence-in-a-postgresql-database.)
9393
* [Creates sequence in PostgreSQL database.](#creates-sequence-in-postgresql-database.)
9494
* [Drops a sequence in PostgreSQL database.](#drops-a-sequence-in-postgresql-database.)
95+
* [Proto-query-data](#proto-query-data)
96+
* [Creates a new database with a proto column and enum](#creates-a-new-database-with-a-proto-column-and-enum)
97+
* [Proto-update-data-dml](#proto-update-data-dml)
98+
* [Proto-update-data](#proto-update-data)
9599
* [Queryoptions](#queryoptions)
96100
* [Quickstart](#quickstart)
97101
* [Read data with database role](#read-data-with-database-role)
@@ -1455,6 +1459,74 @@ __Usage:__
14551459

14561460

14571461

1462+
### Proto-query-data
1463+
1464+
View the [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-query-data.js).
1465+
1466+
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-query-data.js,samples/README.md)
1467+
1468+
__Usage:__
1469+
1470+
1471+
`node samples/proto-query-data.js`
1472+
1473+
1474+
-----
1475+
1476+
1477+
1478+
1479+
### Creates a new database with a proto column and enum
1480+
1481+
View the [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-type-add-column.js).
1482+
1483+
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-type-add-column.js,samples/README.md)
1484+
1485+
__Usage:__
1486+
1487+
1488+
`node proto-type-add-column.js <INSTANCE_ID> <DATABASE_ID> <PROJECT_ID>`
1489+
1490+
1491+
-----
1492+
1493+
1494+
1495+
1496+
### Proto-update-data-dml
1497+
1498+
View the [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-update-data-dml.js).
1499+
1500+
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-update-data-dml.js,samples/README.md)
1501+
1502+
__Usage:__
1503+
1504+
1505+
`node samples/proto-update-data-dml.js`
1506+
1507+
1508+
-----
1509+
1510+
1511+
1512+
1513+
### Proto-update-data
1514+
1515+
View the [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/proto-update-data.js).
1516+
1517+
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-spanner&page=editor&open_in_editor=samples/proto-update-data.js,samples/README.md)
1518+
1519+
__Usage:__
1520+
1521+
1522+
`node samples/proto-update-data.js`
1523+
1524+
1525+
-----
1526+
1527+
1528+
1529+
14581530
### Queryoptions
14591531

14601532
View the [source code](https://github.com/googleapis/nodejs-spanner/blob/main/samples/queryoptions.js).

samples/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
"@google-cloud/kms": "^4.0.0",
1919
"@google-cloud/precise-date": "^4.0.0",
2020
"@google-cloud/spanner": "^7.7.0",
21-
"yargs": "^17.0.0"
21+
"yargs": "^17.0.0",
22+
"protobufjs": "^7.0.0"
2223
},
2324
"devDependencies": {
2425
"chai": "^4.2.0",

samples/proto-query-data.js

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
// Copyright 2024 Google LLC
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
'use strict';
16+
17+
// eslint-disable-next-line node/no-unpublished-require
18+
const singer = require('./resource/singer.js');
19+
const music = singer.examples.spanner.music;
20+
21+
function main(
22+
instanceId = 'my-instance',
23+
databaseId = 'my-database',
24+
projectId = 'my-project-id'
25+
) {
26+
// [START spanner_query_with_proto_types_parameter]
27+
/**
28+
* TODO(developer): Uncomment these variables before running the sample.
29+
*/
30+
// const instanceId = 'my-instance';
31+
// const databaseId = 'my-database';
32+
// const projectId = 'my-project-id';
33+
34+
// Imports the Google Cloud Spanner client library
35+
const {Spanner} = require('@google-cloud/spanner');
36+
37+
// Instantiates a client
38+
const spanner = new Spanner({
39+
projectId: projectId,
40+
});
41+
42+
async function queryDataWithProtoTypes() {
43+
// Gets a reference to a Cloud Spanner instance and database.
44+
const instance = spanner.instance(instanceId);
45+
const database = instance.database(databaseId);
46+
47+
const query = {
48+
sql: `SELECT SingerId,
49+
SingerInfo,
50+
SingerInfo.nationality,
51+
SingerInfoArray,
52+
SingerGenre,
53+
SingerGenreArray
54+
FROM Singers
55+
WHERE SingerInfo.nationality = @country
56+
and SingerGenre=@singerGenre`,
57+
params: {
58+
country: 'Country2',
59+
singerGenre: music.Genre.FOLK,
60+
},
61+
/* `columnsMetadata` is an optional parameter and is used to deserialize the
62+
proto message and enum object back from bytearray and int respectively.
63+
If columnsMetadata is not passed for proto messages and enums, then the data
64+
types for these columns will be bytes and int respectively. */
65+
columnsMetadata: {
66+
SingerInfo: music.SingerInfo,
67+
SingerInfoArray: music.SingerInfo,
68+
SingerGenre: music.Genre,
69+
SingerGenreArray: music.Genre,
70+
},
71+
};
72+
73+
// Queries rows from the Singers table.
74+
try {
75+
const [rows] = await database.run(query);
76+
77+
rows.forEach(row => {
78+
const json = row.toJSON();
79+
console.log(
80+
`SingerId: ${json.SingerId}, SingerInfo: ${json.SingerInfo}, SingerGenre: ${json.SingerGenre},
81+
SingerInfoArray: ${json.SingerInfoArray}, SingerGenreArray: ${json.SingerGenreArray}`
82+
);
83+
});
84+
} catch (err) {
85+
console.error('ERROR:', err);
86+
} finally {
87+
// Close the database when finished.
88+
database.close();
89+
}
90+
}
91+
92+
queryDataWithProtoTypes();
93+
// [END spanner_query_with_proto_types_parameter]
94+
}
95+
96+
process.on('unhandledRejection', err => {
97+
console.error(err.message);
98+
process.exitCode = 1;
99+
});
100+
main(...process.argv.slice(2));

samples/proto-type-add-column.js

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
/**
2+
* Copyright 2024 Google LLC
3+
* Licensed under the Apache License, Version 2.0 (the "License");
4+
* you may not use this file except in compliance with the License.
5+
* You may obtain a copy of the License at
6+
*
7+
* http://www.apache.org/licenses/LICENSE-2.0
8+
*
9+
* Unless required by applicable law or agreed to in writing, software
10+
* distributed under the License is distributed on an "AS IS" BASIS,
11+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
* See the License for the specific language governing permissions and
13+
* limitations under the License.
14+
*/
15+
16+
// sample-metadata:
17+
// title: Creates a new database with a proto column and enum
18+
// usage: node proto-type-add-column.js <INSTANCE_ID> <DATABASE_ID> <PROJECT_ID>
19+
20+
'use strict';
21+
22+
const fs = require('fs');
23+
24+
function main(
25+
instanceId = 'my-instance',
26+
databaseId = 'my-database',
27+
projectId = 'my-project-id'
28+
) {
29+
// [START spanner_add_proto_type_columns]
30+
/**
31+
* TODO(developer): Uncomment the following lines before running the sample.
32+
*/
33+
// const projectId = 'my-project-id';
34+
// const instanceId = 'my-instance-id';
35+
// const databaseId = 'my-database-id';
36+
37+
// Imports the Google Cloud client library
38+
const {Spanner} = require('@google-cloud/spanner');
39+
40+
// Creates a client
41+
const spanner = new Spanner({
42+
projectId: projectId,
43+
});
44+
45+
const databaseAdminClient = spanner.getDatabaseAdminClient();
46+
async function protoTypeAddColumn() {
47+
// Adds a new Proto Message column and Proto Enum column to the Singers table.
48+
49+
const request = [
50+
`CREATE PROTO BUNDLE (
51+
examples.spanner.music.SingerInfo,
52+
examples.spanner.music.Genre,
53+
)`,
54+
'ALTER TABLE Singers ADD COLUMN SingerInfo examples.spanner.music.SingerInfo',
55+
'ALTER TABLE Singers ADD COLUMN SingerInfoArray ARRAY<examples.spanner.music.SingerInfo>',
56+
'ALTER TABLE Singers ADD COLUMN SingerGenre examples.spanner.music.Genre',
57+
'ALTER TABLE Singers ADD COLUMN SingerGenreArray ARRAY<examples.spanner.music.Genre>',
58+
];
59+
60+
// Read a proto descriptor file and convert it to a base64 string
61+
const protoDescriptor = fs
62+
.readFileSync('./resource/descriptors.pb')
63+
.toString('base64');
64+
65+
// Alter existing table to add a column.
66+
const [operation] = await databaseAdminClient.updateDatabaseDdl({
67+
database: databaseAdminClient.databasePath(
68+
projectId,
69+
instanceId,
70+
databaseId
71+
),
72+
statements: request,
73+
protoDescriptors: protoDescriptor,
74+
});
75+
76+
console.log(`Waiting for operation on ${databaseId} to complete...`);
77+
await operation.promise();
78+
console.log(
79+
`Altered table "Singers" on database ${databaseId} on instance ${instanceId} with proto descriptors.`
80+
);
81+
}
82+
protoTypeAddColumn();
83+
// [END spanner_add_proto_type_columns]
84+
}
85+
86+
process.on('unhandledRejection', err => {
87+
console.error(err.message);
88+
process.exitCode = 1;
89+
});
90+
main(...process.argv.slice(2));

0 commit comments

Comments
 (0)