REST API Documentation: Files

Concepts

The File API enables uploading and transforming data for subsequent visualization by the visualization dataset API. It is meant for use cases like reusing the same file across different visualizations, such as for faster interactive data cleaning. Multiple file formats may be uploaded.

The flow is roughly:

POST a file, with potential settings
POST data upload for the file, triggering a parsing attempt
Potentially iterate over sending PATCH calls for the file or POSTs for different data
Create a visualization that uses the uploaded file(s)

List Files

Route Method Headers Parameters Return

Route	Method	Headers	Parameters	Return
`api/v2/files/`	GET	Content-Type: application/json Authorization: Bearer YOUR_JWT_TOKEN	`{ ?"limit": int, ?"offset": int, }`	`{ "count": int, "next": str \| null, "previous": str \| null, "results": [{ "created_at": str, "updated_at": str, "agent_name": str, "agent_version": str, "author": int, "bytes_count": int, "description": str, "file_id": str, "name": str, "file_compression": str, "file_type": str, "parser_options": dict, "tables_schemas": [{ "dtypes": {<str>: str}, "name": str, "num_cols": int, "num_rows": int }] "sql_transforms": json, "is_uploaded": bool, "is_valid": bool, "errors": dict }] }`
	Input: `curl -X GET \ -H "Authorization: Bearer my_generated_token" \ -H "Content-Type: application/json" \ http://localhost/api/v2/files/?limit=100`
	Output: { "count": 1, "next": null, "previous": null, "results": [ { "created_at": "2024-05-17T18:46:14.324431Z", "updated_at": "2024-05-17T18:46:15.047607Z", "agent_name": "", "agent_version": "", "author": 1, "bytes_count": 4, "description": "", "file_id": "e1f99a917cbf42ef981e53ec7ac0aea4", "name": "myfile", "file_compression": "zip", "file_type": "json", "parser_options": {}, "tables_schemas": [ { "name": "filename", "dtypes": { "id": "object" }, "num_cols": 1, "num_rows": 12 } ], "sql_transforms": null, "is_uploaded": true, "is_valid": true, "errors": {} } ] }

api/v2/files/

GET

Content-Type: application/json
Authorization: Bearer YOUR_JWT_TOKEN


{
  ?"limit": int,
  ?"offset": int,
}


{
  "count": int,
  "next": str | null,
  "previous": str | null,
  "results": [{
    "created_at": str,
    "updated_at": str,
    "agent_name": str,
    "agent_version": str,
    "author": int,
    "bytes_count": int,
    "description": str,
    "file_id": str,
    "name": str,
    "file_compression": str,
    "file_type": str,
    "parser_options": dict,
    "tables_schemas": [{
      "dtypes": {<str>: str},
      "name": str,
      "num_cols": int,
      "num_rows": int
    }]
    "sql_transforms": json,
    "is_uploaded": bool,
    "is_valid": bool,
    "errors": dict
  }]
}

Input:

curl -X GET \
  -H "Authorization: Bearer my_generated_token" \
  -H "Content-Type: application/json" \
  http://localhost/api/v2/files/?limit=100

Output:

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "created_at": "2024-05-17T18:46:14.324431Z",
      "updated_at": "2024-05-17T18:46:15.047607Z",
      "agent_name": "",
      "agent_version": "",
      "author": 1,
      "bytes_count": 4,
      "description": "",
      "file_id": "e1f99a917cbf42ef981e53ec7ac0aea4",
      "name": "myfile",
      "file_compression": "zip",
      "file_type": "json",
      "parser_options": {},
      "tables_schemas": [
        {
          "name": "filename",
          "dtypes": {
            "id": "object"
          },
          "num_cols": 1,
          "num_rows": 12
        }
      ],
      "sql_transforms": null,
      "is_uploaded": true,
      "is_valid": true,
      "errors": {}
    }
  ]
}

Create file

Route Method Headers Parameters Return

Route	Method	Headers	Parameters	Return
`api/v2/files/`	POST	Content-Type: application/json Authorization: Bearer YOUR_JWT_TOKEN	`{ ?"org_name": str, "file_type": "arrow" \| "csv" \| "json" \| "orc" \| "parquet" \| "xls" \| "xlsx", ?"name": str, ?"description": str, ?"file_compression": "" \| "gzip" \| "bz2" \| "zip" \| "xz" ?"parser_options": json, ?"sql_transforms": json }`	`{ "created_at": str, "description": str, "file_id": str, "file_type": str, "name": str, }`
	Input: `curl -s -L -X POST \ -H "Authorization: Bearer my_generated_token" \ -H "Content-Type: application/json" \ -d'{"file_type": "csv", "name": "mycsv"}' \ http://localhost/api/v2/files/`
	Output: `{ "created_at": "2021-01-27T05:07:55.971417Z", "description": "", "file_type": "csv", "name": "mycsv" }`

api/v2/files/

POST

Content-Type: application/json
Authorization: Bearer YOUR_JWT_TOKEN


{
  ?"org_name": str,
  "file_type": "arrow" | "csv" | "json" | "orc" | "parquet" | "xls" | "xlsx",
  ?"name": str,
  ?"description": str,
  ?"file_compression": "" | "gzip" | "bz2" | "zip" | "xz"
  ?"parser_options": json,
  ?"sql_transforms": json
}


{
  "created_at": str,
  "description": str,
  "file_id": str,
  "file_type": str,
  "name": str,
}

Input:


curl -s -L -X POST \
  -H "Authorization: Bearer my_generated_token" \
  -H "Content-Type: application/json" \
  -d'{"file_type": "csv", "name": "mycsv"}' \
  http://localhost/api/v2/files/

Output:


{
    "created_at": "2021-01-27T05:07:55.971417Z",
    "description": "",
    "file_type": "csv",
    "name": "mycsv"
}

Upload file data

Route Method Headers Parameters Return

Route	Method	Headers	Parameters	Return
`api/v2/upload/files/<file_id>`	POST	Authorization: Bearer YOUR_JWT_TOKEN	file data	`{ "author": int, "bytes_count": int, "created_at": str, "updated_at": str, "description": str, "errors": dict, "file_compression": str, "file_id": str, "file_type": str, "is_uploaded": bool, "is_valid": bool, "name": str, "parser_options": dict, "tables_schemas": [ { "dtypes": {<str>: str}, "name": str, "num_cols": int, "num_rows": int }] }`
	Input: `curl -s -X POST \ -H "Authorization: Bearer my_generated_token" \ -d '{"s": ["a", "b", "c"], "d": ["b", "c", "a"], "v": [1, 1, 2]}' \ http://localhost/api/v2/upload/files/<file_id>` `curl -s -X POST \ -H "Authorization: Bearer my_generated_token" \ --data-binary "@/home/myfolder/myfile" \ http://localhost/api/v2/upload/files/<file_id>`
	Output: `{ "author": 123, "bytes_count": 4, "created_at": "2021-08-23T17:37:24.569721Z", "updated_at": "2021-08-23T17:37:24.569738Z", "description": "", "errors": {}, "file_compression": "", "file_id": "abc123", "file_type": "json", "is_uploaded": true, "is_valid": true, "name": "myedges", "tables_schemas": [ { "dtypes": { "d": "object", "s": "object", "v": "int64" }, "name": "Untitled 0", "num_cols": 3, "num_rows": 3 } ] }`

api/v2/upload/files/<file_id>

POST

Authorization: Bearer YOUR_JWT_TOKEN

file data


{
  "author": int,
  "bytes_count": int,
  "created_at": str,
  "updated_at": str,
  "description": str,
  "errors": dict,
  "file_compression": str,
  "file_id": str,
  "file_type": str,
  "is_uploaded": bool,
  "is_valid": bool,
  "name": str,
  "parser_options": dict,
  "tables_schemas": [ {
    "dtypes": {<str>: str},
    "name": str,
    "num_cols": int,
    "num_rows": int
  }]
}

Input:


curl -s -X POST \
  -H "Authorization: Bearer my_generated_token" \
  -d '{"s": ["a", "b", "c"], "d": ["b", "c", "a"], "v": [1, 1, 2]}' \
  http://localhost/api/v2/upload/files/<file_id>


  curl -s -X POST \
    -H "Authorization: Bearer my_generated_token" \
    --data-binary "@/home/myfolder/myfile" \
    http://localhost/api/v2/upload/files/<file_id>

Output:


{
  "author": 123,
  "bytes_count": 4,
  "created_at": "2021-08-23T17:37:24.569721Z",
  "updated_at": "2021-08-23T17:37:24.569738Z",
  "description": "",
  "errors": {},
  "file_compression": "",
  "file_id": "abc123",
  "file_type": "json",
  "is_uploaded": true,
  "is_valid": true,
  "name": "myedges",
  "tables_schemas": [
    {
      "dtypes": {
        "d": "object",
        "s": "object",
        "v": "int64"
      },
      "name": "Untitled 0",
      "num_cols": 3,
      "num_rows": 3
    }
  ]
}

Delete file

Route	Method	Headers	Return
`api/v2/files/<file_id>`	DELETE	Content-Type: application/json Authorization: Bearer YOUR_JWT_TOKEN	HTTP code 204
	Input: `curl -s -L -X DELETE \ -H "Authorization: Bearer my_generated_token" \ -H "Content-Type: application/json" \ http://localhost/api/v2/files/1234`
	Output: `204`