Skip to main content

Data & Derivatives

In this step we will extend our server so that we can list models, upload them, and prepare them for viewing.

Data management​

First, let's make sure that our application has a bucket in the Data Management service to store its files in. Typically the bucket would be created just once as part of a provisioning step but in our sample we will implement a helper function that will make sure that the bucket is available, and use it in other parts of the server app.

When creating buckets, it is required that applications set a retention policy for objects stored in the bucket. This cannot be changed at a later time. The retention policy on the bucket applies to all objects stored within. When creating a bucket, specifically set the policyKey to transient, temporary, or persistent.

info

Data Retention Policy

Transient - Objects older than 24 hours are removed automatically.

Temporary - When an object has reached 30 days of age, it is deleted.

Persistent - Available until a user deletes the object.

Let's implement the OSS (Object Storage Service) logic of our server application. Add the following code to the end of the services/aps.js file:

services/aps.js
service.ensureBucketExists = async (bucketKey) => {
    const accessToken = await getInternalToken();
    try {
        await ossClient.getBucketDetails(bucketKey, { accessToken });
    } catch (err) {
        if (err.axiosError.response.status === 404) {
            await ossClient.createBucket(Region.Us, { bucketKey: bucketKey, policyKey: PolicyKey.Persistent }, { accessToken});
        } else {
            throw err;
        }
    }
};

service.listObjects = async () => {
    await service.ensureBucketExists(APS_BUCKET);
    const accessToken = await getInternalToken();
    let resp = await ossClient.getObjects(APS_BUCKET, { limit: 64, accessToken });
    let objects = resp.items;
    while (resp.next) {
        const startAt = new URL(resp.next).searchParams.get('startAt');
        resp = await ossClient.getObjects(APS_BUCKET, { limit: 64, startAt, accessToken });
        objects = objects.concat(resp.items);
    }
    return objects;
};

service.uploadObject = async (objectName, filePath) => {
    await service.ensureBucketExists(APS_BUCKET);
    const accessToken = await getInternalToken();
    const obj = await ossClient.upload(APS_BUCKET, objectName, filePath, { accessToken });
    return obj;
};

The ensureBucketExists function will simply try and request additional information about a specific bucket using the APS SDK, and if the response from APS is 404 Not Found, it will attempt to create a new bucket with this name.

As you can see, the getObjects method (responsible for listing files in a Data Management bucket) uses pagination. In our code we simply iterate through all the pages and return all files from our application's bucket in a single list.

caution

Note that the Data Management service requires bucket names to be globally unique, and attempts to create a bucket with an already used name will fail with 409 Conflict. See the documentation for more details.

Derivatives​

Next, we will implement a couple of helper functions that will derive/extract various types of information from the uploaded files - for example, 2D drawings, 3D geometry, and metadata - that we can later load into the Viewer in our webpage. To do so, we will need to start a new conversion job in the Model Derivative service, and checking the status of the conversion.

Model Derivative service requires all IDs we use in the API calls to be base64-encoded, so we include a small utility function that will help with that.

info

Base64-encoded IDs are referred to as URNs.

Let's implement the logic for converting designs for viewing, and for checking the status of the conversions. Add the following code to the end of the services/aps.js file:

services/aps.js
service.translateObject = async (urn, rootFilename) => {
    const accessToken = await getInternalToken();
    const job = await modelDerivativeClient.startJob({
        input: {
            urn,
            compressedUrn: !!rootFilename,
            rootFilename
        },
        output: {
            formats: [{
                views: [View._2d, View._3d],
                type: OutputType.Svf2
            }]
        }
    }, { accessToken });
    return job.result;
};

service.getManifest = async (urn) => {
    const accessToken = await getInternalToken();
    try {
        const manifest = await modelDerivativeClient.getManifest(urn, { accessToken });
        return manifest;
    } catch (err) {
        if (err.axiosError.response.status === 404) {
            return null;
        } else {
            throw err;
        }
    }
};

service.urnify = (id) => Buffer.from(id).toString('base64').replace(/=/g, '');

Server endpoints​

Now let's make the new functionality available to the client through another set of endpoints.

Create a models.js file under the routes subfolder with the following code:

routes/models.js
const express = require('express');
const formidable = require('express-formidable');
const { listObjects, uploadObject, translateObject, getManifest, urnify } = require('../services/aps.js');

let router = express.Router();

router.get('/api/models', async function (req, res, next) {
    try {
        const objects = await listObjects();
        res.json(objects.map(o => ({
            name: o.objectKey,
            urn: urnify(o.objectId)
        })));
    } catch (err) {
        next(err);
    }
});

router.get('/api/models/:urn/status', async function (req, res, next) {
    try {
        const manifest = await getManifest(req.params.urn);
        if (manifest) {
            let messages = [];
            if (manifest.derivatives) {
                for (const derivative of manifest.derivatives) {
                    messages = messages.concat(derivative.messages || []);
                    if (derivative.children) {
                        for (const child of derivative.children) {
                            messages.concat(child.messages || []);
                        }
                    }
                }
            }
            res.json({ status: manifest.status, progress: manifest.progress, messages });
        } else {
            res.json({ status: 'n/a' });
        }
    } catch (err) {
        next(err);
    }
});

router.post('/api/models', formidable({ maxFileSize: Infinity }), async function (req, res, next) {
    const file = req.files['model-file'];
    if (!file) {
        res.status(400).send('The required field ("model-file") is missing.');
        return;
    }
    try {
        const obj = await uploadObject(file.name, file.path);
        await translateObject(urnify(obj.objectId), req.fields['model-zip-entrypoint']);
        res.json({
            name: obj.objectKey,
            urn: urnify(obj.objectId)
        });
    } catch (err) {
        next(err);
    }
});

module.exports = router;
info

The formidable() middleware used in the POST request handler will make sure that any multipart/form-data content coming with the request is parsed and available in the req.files and req.fields properties.

And mount the router to our server application by modifying server.js:

server.js
const express = require('express');
const { PORT } = require('./config.js');

let app = express();
app.use(express.static('wwwroot'));
app.use(require('./routes/auth.js'));
app.use(require('./routes/models.js'));
app.listen(PORT, function () { console.log(`Server listening on port ${PORT}...`); });

The router will handle 3 types of requests:

  • GET /api/models - when the client wants to get the list of all models available for viewing
  • GET /api/models/:urn/status - used to check the status of the conversion (incl. error messages if there are any)
  • POST /api/models - when the client wants to upload a new model and start its translation

Try it out​

Start (or restart) the app as usual, and navigate to http://localhost:8080/api/models in the browser. The server should respond with a JSON list with names and URNs of all objects available in your configured bucket.

info

If this is your first time working with APS, you may get a JSON response with an empty array ([]) which is expected. In the screenshot below we can already see a couple of files that were uploaded to our bucket in the past.

tip

If you are using Google Chrome, consider installing JSON Formatter or a similar extension to automatically format JSON responses.

Server Response