S3 output bucket

Results are written to an S3 bucket, mounted through the Mountpoint for Amazon S3 CSI driver, as {jobId}/{name} objects.

Create the bucket

Create it in the cluster’s region and keep Block Public Access on. FineParser never deletes results, so add a lifecycle rule if they should expire:

aws s3api create-bucket --region <region> --bucket my-fineparser-output \
  --create-bucket-configuration LocationConstraint=<region>

aws s3api put-bucket-lifecycle-configuration --bucket my-fineparser-output \
  --lifecycle-configuration '{"Rules":[{"ID":"expire","Status":"Enabled","Filter":{},"Expiration":{"Days":30}}]}'

In us-east-1, leave out --create-bucket-configuration.

Install the Mountpoint for Amazon S3 CSI driver

The driver mounts the bucket with its own IAM role. Grant that role this policy:

fineparser-s3-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-fineparser-output"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-fineparser-output/*"
    }
  ]
}

aws iam create-policy --policy-name FineParserOutputBucket \
  --policy-document file://fineparser-s3-policy.json

eksctl create iamserviceaccount \
  --cluster <cluster> --region <region> \
  --namespace kube-system --name s3-csi-driver-sa \
  --role-name AmazonEKS_S3_CSI_DriverRole --role-only \
  --attach-policy-arn arn:aws:iam::<account>:policy/FineParserOutputBucket \
  --approve

aws eks create-addon --region <region> --cluster-name <cluster> \
  --addon-name aws-mountpoint-s3-csi-driver \
  --service-account-role-arn arn:aws:iam::<account>:role/AmazonEKS_S3_CSI_DriverRole

If the driver is already installed, attach the policy to its existing role. For a bucket encrypted with a customer managed KMS key, also allow kms:GenerateDataKey and kms:Decrypt on the key. See Mountpoint for Amazon S3 CSI driver.

Configure the chart

fineparser-values.yaml

storage:
  output:
    s3:
      bucket: my-fineparser-output
  s3:
    region: us-east-2

The default mount options are allow-delete, allow-other, uid=65534, and gid=65534. Setting storage.s3.mountOptions replaces the list, so keep those four. For example, to write under a prefix:

fineparser-values.yaml

storage:
  s3:
    mountOptions: [allow-delete, allow-other, uid=65534, gid=65534, prefix fineparser/]

Using your own claim

Create a PersistentVolume and claim for the bucket, or any other ReadWriteMany volume, and set storage.output.existingClaim:

fineparser-output.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: fineparser-output
spec:
  capacity:
    storage: 10Gi
  accessModes: [ReadWriteMany]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  mountOptions: [allow-delete, allow-other, uid=65534, gid=65534, region us-east-2]
  csi:
    driver: s3.csi.aws.com
    volumeHandle: fineparser-output   # unique across the cluster's Mountpoint volumes
    volumeAttributes:
      bucketName: my-fineparser-output
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fineparser-output
  namespace: fineparser
spec:
  accessModes: [ReadWriteMany]
  storageClassName: ""
  volumeName: fineparser-output
  resources:
    requests:
      storage: 10Gi

fineparser-values.yaml

storage:
  output:
    existingClaim: fineparser-output

Getting Started

Basics

Concepts

Best Practices

AWS

Reference

Create the bucket

Install the Mountpoint for Amazon S3 CSI driver

Configure the chart

Using your own claim

​Create the bucket

​Install the Mountpoint for Amazon S3 CSI driver

​Configure the chart

​Using your own claim

Create the bucket

Install the Mountpoint for Amazon S3 CSI driver

Configure the chart

Using your own claim