AWS Simple Storage Service, or better known as S3, is an object storage service that offers high durability, availability, and performance. Being one of the most frequently used service in AWS, its support and service level agreement offers extremely high value for tech companies who have needs for storage solutions.
Though S3 by itself is excellent, to integrate the S3 solutions with your own product can be tricky at times. This article aims to help explain how to use Node.js to create a simple service that integrates easily with S3.
If you are proficient with node.js and AWS S3 already, you can skip the document by looking at source code directly: https://github.com/fintelics/fintelics-aws-lab/tree/master/nodejs
Here are the middleware and node modules being used:
Multer: middleware being used to help uploading file
AWS-SDK: node module being used to connect to AWS resources
Express.js: node.js framework that provides a robust set of features for web and mobile applications.
Prerequisite
Before starting the tutorial, several prerequisite steps need to be taken:
- Register and sign into an AWS account
- Create a S3 Bucket
- Create a user with the right permission policy attached to it
Now let’s begin
Configure the project skeleton:
Create the app package.json by using npm init. Then run
npm -i aws-sdk multer multer-s3 express body-parser dotenvTo install the correct dependencies
Configure the .env file:
Set up the .env file in the config folder and place your credential information there. As following:
AWS_ACCESS_KEY_ID=Your Key ID HereAWS_SECRET_ACCESS_KEY=Your Key Secret HereAWS_REGION=The Region of Your Object StorageS3_BUCKET=Your Bucket Name
Note that the reason we create .env file is to avoid hardcoding your access credentials in the source code directly as it can be dangerous especially you decide to push this code to a public repository(That is how uber got hacked), then use .gitignore file to ignore the .env file.
Setup configuration helper file
Then create a config.js file to parse all the environment variables to the desired objects. To help organizing the file structure, dotenv is used.
const path = require(‘path’);const configVar = require(‘dotenv’).config({ path: path.join(__dirname, ‘/.env’) });const aws = require(‘aws-sdk’);aws.config.update({// Your SECRET ACCESS KEY from AWS should go here,// Never share it!// Setup Env Variable, e.g: process.env.SECRET_ACCESS_KEYsecretAccessKey: configVar.parsed.AWS_SECRET_ACCESS_KEY,// Not working key, Your ACCESS KEY ID from AWS should go here,// Never share it!// Setup Env Variable, e.g: process.env.ACCESS_KEY_IDaccessKeyId: configVar.parsed.AWS_ACCESS_KEY_ID,region: configVar.parsed.AWS_REGION // region of your bucket});const s3 = new aws.S3();module.exports={aws,s3,s3bucket:configVar.parsed.S3_BUCKET}
S3 Bucket Upload Routes
Now we use multer together with aws-sdk to upload the files to the actual bucket. We also need to create the REST API for the upload process
const {s3,s3bucket} = require(‘../config/config’)const express = require(‘express’)const router = express.Router();const multer = require(‘multer’);const multerS3 = require(‘multer-s3’);router.get(‘/’, function (req, res) {res.sendFile(__dirname + ‘/index.html’);})router.post(‘/upload’,function (req, res) {const multerUpload = multer({storage: multerS3({s3: s3,key: function (req, file, cb) {cb(null, Date.now().toString())}})}).array(‘file’, 1);multerUpload(req, res, function (err) {if (err) {res.send(‘multerUpload Error:’ + err);}else{res.send(“Success”)}})})module.exports = router
Notice that aside from the uploading post request, there’s also a get request, the reason is that this request is used to pass back the frontend html file for the user interface. The frontend html is a simple form that allows user to submit file as input.
<!DOCTYPE html><html><head lang=”en”><meta charset=”UTF-8"><title></title></head><body>You can upload the file to s3 directly here<form method=”post” enctype=”multipart/form-data” action=”/s3upload/upload”><p><input type=”file” name=”file”/></p><p><input type=”submit”/></p></form></body></html>
Program the index.js file
The index.js file needs to contain the right path for the folder structure. It orchestrate the different components built to support the service. References the configuration environment variables, as well as the path to the S3 bucket upload routes.
const express = require(‘express’),bodyParser = require(‘body-parser’);const {aws,s3} = require(‘./config/config’);const s3upload = require(‘./s3-upload/s3-multer’);const app = express();app.use(bodyParser.json());app.use(‘/s3upload’,s3upload)app.listen(3000, function () {console.log(‘app listening on port 3000!’);});
