Executing tasks in parallel - tutorial

This tutorial will lead you through the steps required to execute tasks in parallel on the Golem Network. The example utilize the task-executor library, which is part of the Golem's JS SDK.

We will go through the following steps:

• Define the problem and split it into chunks that can be executed in parallel
• Create a Golem image
• Create a requestor script
• Run the tasks in parallel and process the output

• Yagna service is installed and running with the try_golem app-key configured (instructions).
• Docker installed and Docker service available.

As a practical example of a problem that is suitable for parallel processing, we selected the task of recovering passwords using the hashcat tool. You can apply a similar procedure to utilize Golem Network for other problems that require parallel processing.

Let's assume we have a password hash obtained from an unknown password processed using the "phpass" algorithm. (Phpass is used as a hashing method by popular web frameworks such as WordPress and Drupal) and we know a password mask ?a?a?a (this means the password consists of three alphanumeric characters).

Our objective is to recover the password using Hashcat - a command-line utility that cracks unknown passwords from their known hashes. Hashcat supports 320 hashing algorithms and 5 different attack types, but for this example, we`ll use only the "phpass" algorithm and a simple brute-force attack.

To find the password that matches the given hash and mask, we could run the following command:

hashcat -a 3 -m 400 in.hash ?a?a?a

where

• -a 3 specifies a brute-force attack,
• -m 400 specifies the phpass algorithm,
• in.hash is the name of the file that contains the hashed password (for our examples we will use: $P$5ZDzPE45CLLhEx/72qt3NehVzwN2Ry/) and
• ?a?a?a is the mask to use.

As a result of the above command, the file hashcat.pot file would be created with the following content: $P$5ZDzPE45CLLhEx/72qt3NehVzwN2Ry/:pas. The output consists of the original hash and the recovered password pas after :.

To speed up the process we can split the keyspace of the potential solution into smaller subsets and run the tasks in parallel.

To achieve this we need to determine the size of the keyspace for a given mask and algorithm:

hashcat --keyspace -a 3 ?a?a?a -m 400

The keyspace for this case is 9025 which could be split into 3 segments with the following boundaries:

• 0..3008
• 3009..6017
• 6018..9025

Once we have the segment limits defined we can use --skip and --limit options in the hashcat command:

hashcat -a 3 -m 400 in.hash --skip 3009 --limit 6016  ?a?a?a

The above command will process the 3009..6016 fragment, and any results in that range will be written to the hashcat.pot file.

Create a project folder and open it.

mkdir parallel-example
cd parallel-example

The tasks that we send to the remote computer are executed in the context of the specified software package - the image. When we create the Task Executor we provide the hash of the image that will be used during processing. In the Quickstart example, we used an image that contained Node.js.

In our case, we need to prepare a custom image containing hashcat software that we will use on the provider’s machines. Golem images are converted from Docker images, so we can start with any existing Docker image that meets your needs and modify it to create a custom one. For our task, we will use an off-the-shelf hashcat image (dizcza/docker-hashcat:intel-cpu) and modify it slightly for Golem.

Create a Dockerfile file with the following content:

FROM ubuntu
WORKDIR /golem/work
RUN apt update
RUN apt install -y hashcat

We use ubuntu Docker image as starting point, and then we define a working directory - WORKDIR /golem/entrypoint.

To build the Docker image, use the following commands:

docker build . -t hashcat

If you have not installed it yet, install the Golem image conversion tool (gvmkit-build):

npm install -g @golem-sdk/gvmkit-build

then convert the Docker image into Golem format

gvmkit-build hashcat

and upload the image anonymously to the repository.

gvmkit-build --direct-file-upload hashcat.gvmi --push --nologin

This command will produce the hash of the image that you can use in the example.

....
-- image link (for use in SDK): <here is the image hash ID>
....

Based on the usage of the hashcat tool, our algorithm will be straightforward:

• First we will calculate the keyspace, then
• Split it into several segments and run tasks in parallel on many providers, as defined by the user.
• Finally, we will collect the results and provide the user with the output.

Note, we could calculate the keyspace locally, but in this example we will also do it on a remote computer, avoiding installing hashcat on our computer.

Now initialize the project, and install the @golem-sdk/task-executor library.

npm init
npm install @golem-sdk/task-executor @golem-sdk/pino-logger commander

Create the index.mjs file with the following content:

import { TaskExecutor } from '@golem-sdk/task-executor'
import { pinoPrettyLogger } from '@golem-sdk/pino-logger'
import { program } from 'commander'

async function main(args) {
  // todo: Create Executor
  // todo: Calculate keyspace
  // todo: Calculate boundaries for each chunk
  // todo: Run the task on multiple providers in parallel
  // todo: Process and print results
  // todo: Shutdown executor
}

program
  .option(
    '--number-of-providers <number_of_providers>',
    'number of providers',
    (value) => parseInt(value),
    3
  )
  .option('--mask <mask>')
  .requiredOption('--hash <hash>')
program.parse()
const options = program.opts()
main(options).catch((error) => console.error(error))

We use the commander library to pass arguments such as --mask and --number-of-providers. This library will print a nice argument description and an example invocation when we run the requestor script with --help.

The main function contains the body of the requestor application. Its sole argument, args, contains information on the command-line arguments read by the argument parser.

We do not run anything on Golem yet.

To execute our tasks on the Golem Network, we need to create a TaskExecutor instance.

const executor = await TaskExecutor.create({
  logger: pinoPrettyLogger(),
  api: { key: 'try_golem' },
  demand: {
    workload: {
      imageHash: '2d665b6a73d4a17e2a8e6fc726aab827fcf020cdfac62ec398dd00e4',
      minMemGib: 8,
    },
  },
  market: {
    rentHours: 0.5,
    pricing: {
      model: 'linear',
      maxStartPrice: 0.5,
      maxCpuPerHourPrice: 1.0,
      maxEnvPerHourPrice: 0.5,
    },
  },
  task: {
    maxParallelTasks: args.numberOfProviders,
  },
})

The imageHash option points to the image that we want the containers to run. We use the hash of the image created by us, but you can use the hash received from gvmkit-build when you created your image.

The other constructor parameters are typical configuration parameters the same as in the other examples.

The maxParallelTasks parameter defines how many parts the task will be divided into and how many parallel tasks will be calculated at the same time.

The first step in the computation is to check the keyspace size. For this, we only need to execute hashcat with --keyspace and read the commands' output. With the TaskExecutor instance running, we can now send such a task to one of the providers using the run method:

const keyspace = await executor.run(async (exe) => {
  const result = await exe.run(`hashcat --keyspace -a 3 ${args.mask} -m 400`)
  return parseInt(result.stdout || '')
})

if (!keyspace) throw new Error(`Cannot calculate keyspace`)
console.log(`Keyspace size computed. Keyspace size = ${keyspace}.`)

This call tells the executor to execute a single task defined by the task function async (exe) => {}. The exe object represent the exeUnit on the provider and allows us to run a task on the provider side. The keyspace size is obtained from the stdout attribute of the result object returned by the task function. In case we cannot calculate the size of the keyspace we will throw an error.

As we will run hashcat on a fragment of the whole keyspace, using the --skip and --limit parameters, we need to split the keyspace into chunks. Knowing the keyspace size and maximum number of providers we range for each of the tasks:

const step = Math.floor(keyspace / args.numberOfProviders + 1)
const range = [...Array(Math.floor(keyspace / step) + 1).keys()].map(
  (i) => i * step
)

Note that the number of chunks does not determine the number of engaged providers. In this example, we decided to split the job into 3 tasks, but the number of providers we want to engage is determined by the maxParallelTasks parameter. The executor will try to engage that number of providers and then pass the tasks to them. Once a provider is ready to execute a task, it takes up the next task from a common pool of tasks. As such, a fast provider may end up executing more tasks than a slow one.

Next, we can start looking for the password using multiple workers, executing the tasks on multiple providers simultaneously.

For each task, we perform the following steps:

• Execute hashcat with proper --skip and --limit values on the provider.
• Get the hashcat\_{skip}.potfile from the provider to the requestor.
• Parse the result from the .potfile.

Let's first create a function that will look for the password in a given range of the keyspace.

const findPasswordInRange = async (skip) => {
  const password = await executor.run(async (exe) => {
    const [, potfileResult] = await exe
      .beginBatch()
      .run(
        `hashcat -a 3 -m 400 '${args.hash}' '${args.mask}' --skip=${skip} --limit=${step} -o pass.potfile || true`
      )
      .run('cat pass.potfile || true')
      .end()
    if (!potfileResult.stdout) return false
    // potfile format is: hash:password
    return potfileResult.stdout.toString().trim().split(':')[1]
  })
  if (!password) {
    throw new Error(`Cannot find password in range ${skip} - ${skip + step}`)
  }
  return password
}

Note, that we use the beginBatch() method to organize together two sequential commands: the first will run the hashcat and the second will print the content of the output file. As we conclude the batch with the end() method the task function will return an array of results objects. As the cat pass.potfile is run as a second command its result will be at index 1 so we can use array destructuring to grab only that result. Keep in mind that tasks executed on a single worker instance run within the same virtual machine and share the contents of a VOLUME. It means that files in the VOLUME left over from one task execution will be present in a subsequent run as long as the execution takes place on the same provider and thus, the same file system.

Let's run our function for each of the ranges. We only need to wait for the first successful result so we can use the Promise.any method.

try {
  const password = await Promise.any(range.map(findPasswordInRange))
  console.log(`Password found: ${password}`)
} catch (err) {
  console.log(`Password not found`)
} finally {
  await executor.shutdown()
}

Once we get the password we print it in the console and end executor.

import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { program } from "commander";

async function main(args) {
  const executor = await TaskExecutor.create({
    logger: pinoPrettyLogger(),
    api: { key: "try_golem" },
    demand: {
      workload: {
        imageHash: "055911c811e56da4d75ffc928361a78ed13077933ffa8320fb1ec2db",
      },
    },
    market: {
      rentHours: 0.5,
      pricing: {
        model: "linear",
        maxStartPrice: 0.5,
        maxCpuPerHourPrice: 1.0,
        maxEnvPerHourPrice: 0.5,
      },
    },
    task: {
      maxParallelTasks: args.numberOfProviders,
    },
  });

  const keyspace = await executor.run(async (exe) => {
    const result = await exe.run(`hashcat --keyspace -a 3 ${args.mask} -m 400`);
    return parseInt(result.stdout || "");
  });

  if (!keyspace) throw new Error(`Cannot calculate keyspace`);

  console.log(`Keyspace size computed. Keyspace size = ${keyspace}.`);
  const step = Math.floor(keyspace / args.numberOfProviders);
  const range = [...Array(Math.floor(keyspace / step)).keys()].map((i) => i * step);

  const findPasswordInRange = async (skip) => {
    const password = await executor.run(async (exe) => {
      const [, potfileResult] = await exe
        .beginBatch()
        .run(`hashcat -a 3 -m 400 '${args.hash}' '${args.mask}' --skip=${skip} --limit=${step} -o pass.potfile || true`)
        .run("cat pass.potfile || true")
        .end();
      if (!potfileResult.stdout) return false;
      // potfile format is: hash:password
      return potfileResult.stdout.toString().trim().split(":")[1];
    });
    if (!password) {
      throw new Error(`Cannot find password in range ${skip} - ${skip + step}`);
    }
    return password;
  };

  try {
    const password = await Promise.any(range.map(findPasswordInRange));
    console.log(`Password found: ${password}`);
  } catch (err) {
    console.log(`Password not found`);
  } finally {
    await executor.shutdown();
  }
}

program
  .option("--number-of-providers <number_of_providers>", "number of providers", (value) => parseInt(value), 3)
  .option("--mask <mask>")
  .requiredOption("--hash <hash>");
program.parse();
const options = program.opts();
main(options).catch((error) => console.error(error));

To test our script, copy it into the index.mjs file. Ensure your Yagna service is running and run:

node index.mjs  --mask '?a?a?a' --hash '$P$5ZDzPE45CLLhEx/72qt3NehVzwN2Ry/'

You should see an output similar to the one below.

In this tutorial, we led you through the following steps:

• Custom Golem image creation
• Parallel task execution across multiple providers
• Submitting multiple command sequences as a single task
• Reading output from commands executed on a provider