JS Task API Examples: accessing the internet
Introduction
In this article, we will present methods that let you access the Internet.
Outbound feature
For the requestor to be able to use the outbound feature, initiate a connection to from the provider to an external url (target_url
), the following minimal conditions must be met:
- The requestor must request the outbound feature in the demand and include a Computation Manifest there. The manifest must declare the
target_url
. - The provider must offer the
outbound
service at least for thetarget_url
. (So either he needs to offer theoutbound
for any (unrestricted) URLs or thetarget_url
is included in the whitelist).
You can find more information about the feature and the manifest in the following articles: Accessing the Internet and Payload Manifest.
The requestor is responsible for:
- Manifest creation
- Defining demand for outbound access
Both examples are accessing urls included in the default whitelist. Click here to view the whitelist.
Prerequisites
Yagna service is installed and running with try_golem
app-key configured.
How to run examples
Create a project folder, initialize a Node.js project, and install libraries.
mkdir golem-example
cd golem-example
npm init
npm i @golem-sdk/task-executor
npm i @golem-sdk/pino-logger
Next, install Golem SDK CLI
- a companion tool that will facilitate manifest creation.
npm install -g @golem-sdk/cli
To run the examples provided below, copy the code into the index.mjs
file in the project folder and run:
node index.mjs
Basic outbound access
In this example, we will download a file from ipfs.io
. The opfs.io
domain is one of the domains included in the whitelist.
Manifest creation
To create a new manifest run:
golem-sdk manifest create golem/examples-outbound:latest
This will create a basic manifest.json
file. You will use it to inform the provider what GVMI image we will be using. The manifest contains also your application version, application name, and description, all read from your package.json
file (you can edit this information if you want).
Adding outbound configuration
The next step is to configure our manifest, so you can access a public URL. The CLI also has a handy command that will take care of that for you:
golem-sdk manifest net add-outbound https://ipfs.io
This has added 'https://ipfs.io' as the URL you want to access from the provider node. The command can be run multiple times to add more URLs or you can pass them all at once.
Your manifest is ready and stored in the manifest.json
file.
Defining demand for outbound access
The example below demonstrates how to define the demand that will get access to the Internet.
Here's the manifest
{
"version": "0.1.0",
"createdAt": "2023-10-19T11:23:08.156+02:00",
"expiresAt": "2024-01-17T11:23:08.156+01:00",
"metadata": {
"name": "outbound-docs",
"description": "",
"version": "1.0.0"
},
"payload": [
{
"platform": {
"os": "linux",
"arch": "x86_64"
},
"hash": "sha3:dad8f776b0eb9f37ea0d63de42757034dd085fe30cc4537c2e119d80",
"urls": [
"http://registry.golem.network/download/f37c8ba2b534ca631060fb8db4ac218d3199faf656aa2c92f402c2b700797c21"
]
}
],
"compManifest": {
"version": "0.1.0",
"net": {
"inet": {
"out": {
"urls": ["https://github.com", "https://ipfs.io"],
"protocols": ["https"]
}
}
}
}
}
And the requestor code:
import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { readFile } from "fs/promises";
// The example is using url from domain that is included in the outbound Whitelist.
// See https://github.com/golemfactory/ya-installer-resources/tree/main/whitelist for the current default whitelist.
const url = "https://ipfs.io/ipfs/bafybeihkoviema7g3gxyt6la7vd5ho32ictqbilu3wnlo3rs7ewhnp7lly";
(async function main() {
// Load the manifest.
const manifest = await readFile(`./manifest.json`);
// Create and configure a TaskExecutor instance.
const executor = await TaskExecutor.create({
logger: pinoPrettyLogger(),
api: { key: "try_golem" },
demand: {
workload: {
capabilities: ["inet", "manifest-support"],
manifest: manifest.toString("base64"),
},
},
market: {
rentHours: 0.5,
pricing: {
model: "linear",
maxStartPrice: 0.5,
maxCpuPerHourPrice: 1.0,
maxEnvPerHourPrice: 0.5,
},
},
});
try {
await executor.run(async (exe) => {
const result = await exe.run(`curl ${url} -o /golem/work/example.jpg`);
console.log((await exe.run("ls -l")).stdout);
if (result.result === "Ok") {
console.log("File downloaded!");
} else {
console.error("Failed to download the file!", result.stderr);
}
});
} catch (err) {
console.error("The task failed due to", err);
} finally {
await executor.shutdown();
}
})();
Note the most important part:
// Load the manifest file.
const manifest = await readFile(`./manifest.json`)
// Create and configure a TaskExecutor instance.
const executor = await TaskExecutor.create({
logger: pinoPrettyLogger(),
api: { key: 'try_golem' },
demand: {
workload: {
capabilities: ['inet', 'manifest-support'],
manifest: manifest.toString('base64'),
},
},
// ... market options
})
First, in the workload
object we specify additional requirements:
- 'inet' - indicates the requestor requires outbound service from providers,
- 'manifest-support' - indicates that the requestor uses a manifest to specify a demand.
Instead of providing an image tag or hash as in the quickstart example, the manifest
is provided. In the manifest
body listed above, you can see it contains the image hash to indicate the image to be run on providers ( in payload
object ) and a list of urls of domains to which the requestor needs access (in compManifest
object).
Please note the loaded manifest is encoded to base64.
api: { key: 'try_golem' }
- defined the api key, to get access to the Yagna service. This particular key is available if you start the yagna according to the procedure provided in the installation example, you can also configure your own unique keys. See here for instructions.
The rest of the requestor script follows the pattern used in the quickstart example. Once you get the exeUnit
instance you can initiate a connection to the external locations, that are in the domain that was declared in the manifest:
...
const url = "https://ipfs.io/ipfs/bafybeihkoviema7g3gxyt6la7vd5ho32ictqbilu3wnlo3rs7ewhnp7lly";
...
const result = await exe.run(`curl ${url} -o /golem/work/example.jpg`);
Using outbound to install node.js packages
Note: This example shows how to use the outbound whitelist to install a npm package on a provider. It is recommended to install additional packages in a directory that is defined as VOLUME
in the image definition, to avoid filesystem capacity limits. If you have a large number of packages you should rather install them during the image build phase - you will avoid installing them on each provider separately and get your providers ready in a shorter time.
Here's the manifest:
{
"version": "0.1.0",
"createdAt": "2023-11-28T12:23:33.435+01:00",
"expiresAt": "2024-02-26T12:23:33.435+01:00",
"metadata": {
"name": "npm_install",
"description": "Example project",
"version": "1.0.0"
},
"payload": [
{
"platform": {
"os": "linux",
"arch": "x86_64"
},
"hash": "sha3:3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd",
"urls": [
"http://registry.golem.network/v1/image/download?hash=3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd"
]
}
],
"compManifest": {
"version": "0.1.0",
"net": {
"inet": {
"out": {
"urls": ["https://registry.npmjs.org"],
"protocols": ["https"]
}
}
}
}
}
And the requestor code:
import { TaskExecutor } from "@golem-sdk/task-executor";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
import { readFile } from "fs/promises";
const manifest = await readFile(`./manifest_npm_install.json`);
(async function main() {
const executor = await TaskExecutor.create({
logger: pinoPrettyLogger(),
api: { key: "try_golem" },
demand: {
workload: {
manifest: manifest.toString("base64"),
capabilities: ["inet", "manifest-support"],
},
expirationSec: 60 * 30, //30 min
},
market: {
rentHours: 0.5,
pricing: {
model: "linear",
maxStartPrice: 0.5,
maxCpuPerHourPrice: 1.0,
maxEnvPerHourPrice: 0.5,
},
},
// Control the execution of tasks
task: {
maxTaskRetries: 0,
taskTimeout: 120 * 60 * 1000,
},
});
try {
await executor.run(async (exe) => {
console.log("working on provider: ", exe.provider.id);
console.log((await exe.run("npm install moment")).stdout);
console.log((await exe.run(`cat ./package.json`)).stdout);
});
console.log("task completed");
} catch (err) {
console.error("Running the task on Golem failed due to", err);
} finally {
await executor.shutdown();
}
})();