Posted by Salmaan Rashid, Technical Account Manager, Google Cloud Platform
The gcloud command line tool is your gateway to manage and interact with
Google Cloud Platform. Being a command line tool, you're probably already thinking of using system tools like
cat|sed|awk|grep|cut to extract out all the info gcloud offers. In fact, gcloud itself offers a variety of options that will help you avoid having to use those commands. In this article, we describe a couple of options you can use to automatically parse and format the results. We’ll also show you a how to chain these commands together in a bash or powershell script to extract the embedded data.
We’re going to demonstrate three gcloud features which you can extend and combine in a variety of ways:
- filters to return a subset of the result
- format to change how that data is rendered
- projections to apply transforms or logic directly to the data returned
Format
Let's start off by formatting a simple command that you are already familiar with that lists the projects to which you have access:
1.
gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
canvas-syntax-130823 scesproject2 346904393285
windy-bearing-129522 scesproject1 222844913538
Now let’s see the raw out output of this command by asking for the raw JSON format of the response:
2.
gcloud projects list --format="json"
[
{
"createTime": "2016-04-28T22:33:12.274Z",
"labels": {
"env": "test",
"version": "alpha"
},
"lifecycleState": "ACTIVE",
"name": "scesproject1",
"parent": {
"id": "297814986428",
"type": "organization"
},
"projectId": "windy-bearing-129522",
"projectNumber": "222844913538"
},
{
"createTime": "2016-05-11T03:08:13.359Z",
"labels": {
"env": "test",
"version": "beta"
},
"lifecycleState": "ACTIVE",
"name": "scesproject2",
"parent": {
"id": "297814986428",
"type": "organization"
},
"projectId": "canvas-syntax-130823",
"projectNumber": "346904393285"
}
]
Seeing the raw JSON now lets us select the resources we're interested in and the formats we'd like. Let's display the same response in a formatted box sorted by
createdTime and only select certain properties to display:
3.
gcloud projects list --format="table[box,title='My Project List'](createTime:sort=1,name,projectNumber,projectId:label=ProjectID,parent.id:label=Parent)"
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ My Project List │
├──────────────────────────┬──────────────┬────────────────┬──────────────────────┬──────────────┤
│ CREATE_TIME │ NAME │ PROJECT_NUMBER │ ProjectID │ Parent │
├──────────────────────────┼──────────────┼────────────────┼──────────────────────┼──────────────┤
│ 2016-04-28T22:33:12.274Z │ scesproject1 │ 222844913538 │ windy-bearing-129522 │ 297814986428 │
│ 2016-05-11T03:08:13.359Z │ scesproject2 │ 346904393285 │ canvas-syntax-130823 │ 297814986428 │
└──────────────────────────┴──────────────┴────────────────┴──────────────────────┴──────────────┘
Tip: you can derive the JSON path value for a property by using
--format='flattened' flag.
Say you don't want a formatted box, just a table without a border with a simple display of the date property in the format year-month-day:
4.
gcloud projects list --format="table(createTime.date('%Y-%m-%d'),name,projectNumber,projectId)"
CREATE_TIME NAME PROJECT_NUMBER PROJECT_ID
2016-05-11 scesproject2 346904393285 canvas-syntax-130823
2016-04-28 scesproject1 222844913538 windy-bearing-129522
Now let's do some more complex formatting. To see this, list out the Compute Engine zones and peek at the JSON:
5.
gcloud compute zones list --format="json"
{
"creationTimestamp": "2014-05-30T18:35:16.514-07:00",
"description": "us-central1-a",
"id": "2000",
"kind": "compute#zone",
"name": "us-central1-a",
"region": "us-central1",
"selfLink": "https://www.googleapis.com/compute/v1/projects/windy-bearing-129522/
zones/us-central1-a",
"status": "UP"
},
Note the selfLink. It's the fully qualified name that you'd like to parse.
gcloud can help here too by giving you functions to select the JSON value and then extract and parse it. Let’s grab the last part of the URL segment of selfLink by using the
selfLink.scope() function:
6.
gcloud compute zones list --format="value(selfLink.scope())"
us-central1-a
Alternatively, you can extract the value using .basename():
7.
gcloud compute zones list --format="value(selfLink.basename())"
us-central1-a
Suppose you want to extract part of the selfLink starting from the /projects path:
8.
gcloud compute zones list --format="value(selfLink.scope(projects))"
windy-bearing-129522/zones/us-central1-a
Some GCP objects have multi-valued resources and we often need to enumerate them. For example, consider listing out all scopes enabled for a given GCE instance:
9.
gcloud compute instances list --format="json"
"serviceAccounts": [
{
"email": "[email protected]",
"scopes": [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/cloud.useraccounts.readonly"
]
}
],
What we actually want to do here is flatten the multi-valued resources:
10.
gcloud compute instances list --format="flattened(name,serviceAccounts[].email,serviceAccounts[].scopes[].basename())"
name: instance-1
serviceAccounts[0].email: [email protected]
serviceAccounts[0].scopes[0]: devstorage.read_only
serviceAccounts[0].scopes[1]: logging.write
serviceAccounts[0].scopes[2]: monitoring.write
serviceAccounts[0].scopes[3]: cloud.useraccounts.readonly
Or flatten multi-values to a separate line per value:
11.
gcloud compute instances list --filter=name:instance-1 --flatten="serviceAccounts[].scopes[]" --format="csv(name,id,serviceAccounts.email,serviceAccounts.scopes.basename())"
name,id,email,scopes
instance-1,763097360168409044,[email protected],
devstorage.read_only
instance-1,763097360168409044,[email protected],
logging.write
instance-1,763097360168409044,[email protected],
monitoring.write
instance-1,763097360168409044,[email protected],
servicecontrol
instance-1,763097360168409044,[email protected],
service.management
Here is the same information in an easy-to-read, structured format:
12.
gcloud compute instances list --filter=name:instance-1 --format="table[box,no-heading](name,id,serviceAccounts:format='table[box,no-heading](email,scopes:format="table[box,no-heading](.)")')"
┌────────────┬────────────────────┐
│ instance-1 │ 763097360168409044 │
└────────────┴────────────────────┘
┌────────────────────────────────────────────────────┐
│ [email protected]│
└────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────┐
│ https://www.googleapis.com/auth/devstorage.read_only │
│ https://www.googleapis.com/auth/logging.write │
│ https://www.googleapis.com/auth/monitoring.write │
│ https://www.googleapis.com/auth/servicecontrol │
│ https://www.googleapis.com/auth/service.management │
└──────────────────────────────────────────────────────┘
The final formatting example parses a multi-valued resource to display the service account keys with the service account for the following raw output:
13.
gcloud beta iam service-accounts keys list --iam-account [email protected] --project mineral-minutia-820 --format="json"
[
{
"name": "projects/mineral-minutia-820/serviceAccounts/svc-2-429@mineral
-minutia-820.iam.gserviceaccount.com/keys/
04bd2d56d0cc5746b125d17f95d4b0dd654accca",
"validAfterTime": "2016-03-11T05:30:04.000Z",
"validBeforeTime": "2026-03-09T05:30:04.000Z"
},
{
"name": "projects/mineral-minutia-820/serviceAccounts/svc-2-
[email protected]/keys/
1deb44e2f54328fc7bb316e5a87315e3314f114f",
"validAfterTime": "2016-01-02T18:54:26.000Z",
"validBeforeTime": "2025-12-30T18:54:26.000Z"
},
....
]
So use
.scope() to extract just the serviceAccount part, then grab the first '/' delimited part with
segment(0):
14.
gcloud beta iam service-accounts keys list --iam-account [email protected] --project mineral-minutia-820 --format="table(name.scope(serviceAccounts).segment(0):label='service Account',name.scope(keys):label='keyID',validAfterTime)"
 |
| (click to enlarge) |
Filters
Let's talk about filters. Filters allow you to select only the resources to which you want to apply formatting.For example, suppose you labeled your resource (projects, VM's, etc.) with a specific name, and you want to list only those projects where the labels match specific values (e.g.
label.env='test' and
label.version=alpha):
15.
gcloud projects list --format="json" --filter="labels.env=test AND labels.version=alpha"
[
{
"createTime": "2016-04-28T22:33:12.274Z",
"labels": {
"env": "test",
"version": "alpha"
},
"lifecycleState": "ACTIVE",
"name": "scesproject1",
"parent": {
"id": "297814986428",
"type": "organization"
},
"projectId": "windy-bearing-129522",
"projectNumber": "222844913538"
}
]
You can also apply projections on keys. In the example below, the filter is applied on the
createTime key after the date formatting is set:
16.
gcloud projects list --format="table(projectNumber,projectId,createTime)" --filter="createTime.date('%Y-%m-%d', Z)='2016-05-11'"
PROJECT_NUMBER PROJECT_ID CREATE_TIME
346904393285 canvas-syntax-130823 2016-05-11T03:08:13.359Z
Notice the filter selected above actually references a JSON structure (labels.
env=test).You can of course use that and combine it in any number of ways.
Projection transforms
Projection transforms allow you to alter the value rendered directly. We already showed several examples above (e.g.,
.extract(), .scope(), .basename(), .segment()). To note, one interesting capability of transforms is that you can combine and chain them together with .map() and and apply them to multi-valued data.
For example, the following applies conditional projection to the
parent.id key such that if the
parent.id key exists, the output is "YES" and otherwise its "NO". This is a quick way to see which of your projects meets a specific criteria (in this case, is it part of the Organization Node)
17.
gcloud projects list --format="table(projectId,parent.id.yesno(yes="YES", no=”NO”):label='Has Parent':sort=2)"
PROJECT_ID Has Parent
mineral-minutia-820 NO
fabled-ray-104117 YES
rk-test-0506 YES
user2proj1 YES
user2project2 YES
18.
gcloud compute instances list --format="flattened(name,serviceAccounts[].email,serviceAccounts[].scopes.map().scope())"
name: instance-1
serviceAccounts[0].email: [email protected]
serviceAccounts[0].scopes[0]: devstorage.read_only
serviceAccounts[0].scopes[1]: logging.write
serviceAccounts[0].scopes[2]: monitoring.write
serviceAccounts[0].scopes[3]: cloud.useraccounts.readonly
Scripts
Finally, let's see how we can combine
gcloud commands into a script that will help us easily extract embedded information. In the following example, we list all the keys associated with all your projects’ service accounts.To do this, we first need to enumerate all the projects, then for each project, get all of its service accounts.Finally, for each service account, we list all the keys created against it. This is basically a nested loop to iterate over:
As a bash script:
#!/bin/bash
for project in $(gcloud projects list --format="value(projectId)")
do
echo "ProjectId: $project"
for robot in $(gcloud beta iam service-accounts list --project $project --format="value(email)")
do
echo " -> Robot $robot"
for key in $(gcloud beta iam service-accounts keys list --iam-account $robot --project $project --format="value(name.basename())")
do
echo " $key"
done
done
done
Or as Windows PowerShell:
foreach ($project in gcloud projects list --format="value(projectId)")
{
Write-Host "ProjectId: $project"
foreach ($robot in gcloud beta iam service-accounts list --project $project --format="value(email)")
{
Write-Host " -> Robot $robot"
foreach ($key in gcloud beta iam service-accounts keys list --iam-account $robot --project $project --format="value(name.basename())")
{
Write-Host " $key"
}
}
}
You'll also often need to parse response fields into arrays for processing. The following example parses the service account information associated with an instance into an array for easy manipulation. Notice the
serviceAccounts[].scope field is multi-valued within the csv and delimited by a semicolon since we defined "separator=;". That is, each response line from the gcloud command below will be in the form
name,id,email,scope_1;scope_2;scope_3. The script below essentially parses the response from example 12 above:
#!/bin/bash
for scopesInfo in $(
gcloud compute instances list --filter=name:instance-1
--format="csv[no-heading](name,id,serviceAccounts[].email.list(),
serviceAccounts[].scopes[].map().list(separator=;))")
do
IFS=',' read -r -a scopesInfoArray<<< "$scopesInfo"
NAME="${scopesInfoArray[0]}"
ID="${scopesInfoArray[1]}"
EMAIL="${scopesInfoArray[2]}"
SCOPES_LIST="${scopesInfoArray[3]}"
echo "NAME: $NAME, ID: $ID, EMAIL: $EMAIL"
echo ""
IFS=';' read -r -a scopeListArray<<< "$SCOPES_LIST"
for SCOPE in "${scopeListArray[@]}"
do
echo " SCOPE: $SCOPE"
done
done
Hopefully, this has given you ideas for how to effectively filter and format
gcloud command output. You can apply these techniques and extend them to any
gcloud response
— just look at the raw response, think about what you want to do, and then format away!
