Files

Concepts

The File API enables uploading and transforming data for subsequent visualization by the visualization dataset API. It is meant for use cases like reusing the same file across different visualizations, such as for faster interactive data cleaning. Multiple file formats may be uploaded.

The flow is roughly:

  • POST a file, with potential settings
  • POST data upload for the file, triggering a parsing attempt
  • Potentially iterate over sending PATCH calls for the file or POSTs for different data
  • Create a visualization that uses the uploaded file(s)

List Files

Route Method Headers Parameters Return
api/v2/files/ GET Content-Type: application/json
Authorization: Bearer YOUR_JWT_TOKEN

{
  ?"limit": int,
  ?"offset": int,
}
        

{
  "count": int,
  "next": str | null,
  "previous": str | null,
  "results": [{
    "created_at": str,
    "updated_at": str,
    "agent_name": str,
    "agent_version": str,
    "author": int,
    "bytes_count": int,
    "description": str,
    "file_id": str,
    "name": str,
    "file_compression": str,
    "file_type": str,
    "parser_options": dict,
    "tables_schemas": [{
      "dtypes": {<str>: str},
      "name": str,
      "num_cols": int,
      "num_rows": int
    }]
    "sql_transforms": json,
    "is_uploaded": bool,
    "is_valid": bool,
    "errors": dict
  }]
}
        
Input:
curl -X GET \
  -H "Authorization: Bearer my_generated_token" \
  -H "Content-Type: application/json" \
  http://localhost/api/v2/files/?limit=100
Output:
{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "created_at": "2024-05-17T18:46:14.324431Z",
      "updated_at": "2024-05-17T18:46:15.047607Z",
      "agent_name": "",
      "agent_version": "",
      "author": 1,
      "bytes_count": 4,
      "description": "",
      "file_id": "e1f99a917cbf42ef981e53ec7ac0aea4",
      "name": "myfile",
      "file_compression": "zip",
      "file_type": "json",
      "parser_options": {},
      "tables_schemas": [
        {
          "name": "filename",
          "dtypes": {
            "id": "object"
          },
          "num_cols": 1,
          "num_rows": 12
        }
      ],
      "sql_transforms": null,
      "is_uploaded": true,
      "is_valid": true,
      "errors": {}
    }
  ]
}

Create file

Route Method Headers Parameters Return
api/v2/files/ POST Content-Type: application/json
Authorization: Bearer YOUR_JWT_TOKEN

{
  ?"org_name": str,
  "file_type": "arrow" | "csv" | "json" | "orc" | "parquet" | "xls" | "xlsx",
  ?"name": str,
  ?"description": str,
  ?"file_compression": "" | "gzip" | "bz2" | "zip" | "xz"
  ?"parser_options": json,
  ?"sql_transforms": json
}
        

{
  "created_at": str,
  "description": str,
  "file_id": str,
  "file_type": str,
  "name": str,
}
        
Input:

curl -s -L -X POST \
  -H "Authorization: Bearer my_generated_token" \
  -H "Content-Type: application/json" \
  -d'{"file_type": "csv", "name": "mycsv"}' \
  http://localhost/api/v2/files/
Output:

{
    "created_at": "2021-01-27T05:07:55.971417Z",
    "description": "",
    "file_type": "csv",
    "name": "mycsv"
}

Upload file data

Route Method Headers Parameters Return
api/v2/upload/files/<file_id> POST Authorization: Bearer YOUR_JWT_TOKEN file data

{
  "author": int,
  "bytes_count": int,
  "created_at": str,
  "updated_at": str,
  "description": str,
  "errors": dict,
  "file_compression": str,
  "file_id": str,
  "file_type": str,
  "is_uploaded": bool,
  "is_valid": bool,
  "name": str,
  "parser_options": dict,
  "tables_schemas": [ {
    "dtypes": {<str>: str},
    "name": str,
    "num_cols": int,
    "num_rows": int
  }]
}
        
Input:

curl -s -X POST \
  -H "Authorization: Bearer my_generated_token" \
  -d '{"s": ["a", "b", "c"], "d": ["b", "c", "a"], "v": [1, 1, 2]}' \
  http://localhost/api/v2/upload/files/<file_id>

  curl -s -X POST \
    -H "Authorization: Bearer my_generated_token" \
    --data-binary "@/home/myfolder/myfile" \
    http://localhost/api/v2/upload/files/<file_id>
Output:

{
  "author": 123,
  "bytes_count": 4,
  "created_at": "2021-08-23T17:37:24.569721Z",
  "updated_at": "2021-08-23T17:37:24.569738Z",
  "description": "",
  "errors": {},
  "file_compression": "",
  "file_id": "abc123",
  "file_type": "json",
  "is_uploaded": true,
  "is_valid": true,
  "name": "myedges",
  "tables_schemas": [
    {
      "dtypes": {
        "d": "object",
        "s": "object",
        "v": "int64"
      },
      "name": "Untitled 0",
      "num_cols": 3,
      "num_rows": 3
    }
  ]
}

Delete file

Route Method Headers Parameters Return
api/v2/files/<file_id> DELETE Content-Type: application/json
Authorization: Bearer YOUR_JWT_TOKEN
HTTP code 204
Input:

curl -s -L -X DELETE \
  -H "Authorization: Bearer my_generated_token" \
  -H "Content-Type: application/json" \
  http://localhost/api/v2/files/1234
Output:

204