Tag Archives: gcloud

Makefile – Get Compute Engine Instance IP

I have a test that requires me to spin up a VM with a service running on port 80. I’d need to test if that service is working, but the machine itself is irrelevant, so I’m not bothering to give it a hostname. But I still need to test output on port 80. So I need the external IP address of the machine and nothing else.

Enter the dynamic data trick from my first Makefile post – again.  

So I grab the value and dump it to a variable:

Nothing more to it.

Makefile – Start, Stop or Delete 20 VMs at once

Last time I created 20 virtual machines at once.  Now I want to stop those machines, or start them back up, or delete them. Basically, I want to do bulk operations on all of the machines that I am using in this scenario.

If you look at the create 20 VMs post, I gave each one of them a similar name, based on the pattern “load-xxx” where load is the operation I am using them for and xxx is a three digit sequential id with 0s prefixed. (This makes them order correctly in our UI.)

Because I know their names, I can count them up and not have to explicitly tell these operations how many machines I want to operate on.  To do that, I create a make variable that contains the count of all VMs prefixed by “load.”

Once I have that, I can perform batch operations very simply.

To stop 20 running VMs:

Just to explain, like the previous post, we loop from i to COUNT, creating a variable that contains the name of our server, and running a function call to execute the gcloud stop instances command.  Why is this a separate function?  Because I usually do more than just stop the VM.

I also wrap the call in parentheses and append the & to allow multiple calls to execute in parallel.

To start them back up:

To delete them all:

And in this case, I do a little bit more here in delete.  I make sure all of the disks are deleted, and I set the request to quiet. Why? Because I don’t want to confirm this 20 times, silly.

In any case, doing batch operations on my set of VMs is as easy as:

There you have it, fleets of VMs responding in concert to your requests.  As it should be.

Makefile – Launch 20 Compute Engine virtual machines at once.

We’re going to try something a lot more complex in make now. I’m going to dynamically create 20 Compute Engine virtual machines that are absolutely the same. This requires quite a bit more complexity, so we’ll break it down step by step.

Let’s start with the gcloud command to create an instance.  

I encapsulated this into a Makefile function. Why?  Well, as I have it here, it is a pretty simple event with adding apt-get update but I usually do more then just create the node and install software. I often set environmental information or start services, etc. So by putting all of the instance specific instructions in a function, I make it just slightly easier to grok.

Let’s go through this part step by step.  

  • Define a function with the define keyword, and end it with the endef keyword
  • It appears that functions must be one line, so use ;\ to organize multiple calls into one function
  • Wrap all of the real work in a parenthesis. Why? It turns it into one operation, so that each step of the function doesn’t block parallel execution of other operations in the makefile.
  • Capture the first argument – $(1) – passed into this function – we’ll use it as the name of the instance
  • Create a machine using gcloud compute instances create. Note setting the machine type.  If you are creating a lot of instances, make sure you don’t run afoul of quota or spend.
  • SSH into machine and run apt-get update.
  • Tell us this machine is ready.   

Okay, that handles the instance creation, but now we have to loop through and create a variable amount of machines. I said 20, but I often spin up anywhere from 10 to 150 using this method.

Again, step by step:

  • Use @ so that the commands aren’t echoed to the output.
  • Set up a while loop with iterator – i, that will run as long as i is less than the explicitly passed variable named count
  • Use ;\ to make the command one logical line.
  • Use printf to create a variable named server to name the instances. In this case each instance is named “load-xxx” where xxx is a sequential id number for the node that always has three digits. This makes it easier to go back later and do more group operations on the entire set of machines. 
  • Call the function using the syntax $(call function_namevalue_to_pass)
  • Wrap call in parentheses and append a &.  This shoves the call to the background so you can create 20, or 100, or 150 of these in parallel instead of sequentially.
  • We then increment the counter.   

Finally we call the whole thing with:

Pretty straightforward. I frequently use this technique to launch of fleet of VMs to send large amounts of load at App Engine. Next I’ll tell you how to delete them all.   

Don’t forget the count=N, or the call will bail.

Makefile – Clean App Engine flexible environment

One of the more interesting quirks of App Engine flexible environment is that App Engine launches Compute Engine virtual machines that you can’t spin down directly. The way to spin down App Engine flex is to delete all versions of the app.  This will close down all of the VMs, and shut down your App Engine app.

You can do it manually through the web interface, you can do it manually by listing versions in gcloud then deleting them, or you can have a Makefile do it for you.

First I use the trick I wrote about capturing dynamic data from gcloud. Then I pipe that to a Makefile command that will delete the versions.

Note that I add -q to the command because I don’t want to be prompted; I just want them gone.

Makefile – Get dynamic values from gcloud

Most of the time when I create something in my environment on Google Cloud Platform, I give it a specific name.  For example, I create servers and call them “ThingIWillReferenceLaterWhenIDeleteYou” or more boringly, “Server1.”

Having set names, as I alluded to, makes it easier to clean up after yourself. But there are some cases when you cannot name things when they are created. So it would be nice to get a list of these names. For example, App Engine flexible environment versions for cleaning up after a test.

You can get a list of them with this command:

Which yields this:

Now normally I would have to add extra code to my Makefile to rip out the version names.

But gcloud actually has a robust formatting tool. So instead of running the command above I can run:

And get the JSON representation, which looks like this:

Using a JSON parser might make this easier, but there is an even easier way:

Which yields:

What will this do?

It will list just the value of version.id and it will separate each record it returns with a ” “, not a line break.  This allows me to drop this generated list into any command that takes multiple names and run them. The gcloud CLI takes multiple arguments in this way. 

So to make this applicable to Makefiles I have to do one more thing – take this data and put it in a variable.

Here we are, ready to use this variable in other Make commands. This works for most of the other places in GCP where you see random values spitting out, like IP forwarding rules, and GKE nodes, to name two. 

To learn more about how to filter and format your gcloud commands, check out the Google Cloud Platform Blog.