Resources · Knowledge base
Knowledge base
Learning resources to get you started using Eightwire, for administrators and end-users
81 articles · 22 topics
81 articles · 22 topics
Topics
Set up your Account5 articles
- Account Setup & VerificationThis section shows you how to get started using Eightwire by following this process to self-register for an Eightwire Account. The Account is created as a trial which is in place for two weeks, so you can get familiar with Eightwire and set up some test data transfers. Your Account switches to a full license after two weeks or when you are ready to work with production data.
- Account SettingsAccount settings are where you maintain contact details for your account, setup two-factor authentication for your users and manage the time zone that your account uses.
- User SecurityLearn how to tailor access for each member of your Account. Using Role-based security, each user is assigned a combination of account and project-level permissions. This page details the permissions assigned by Account and Project Role.
- Create UsersLearn how to invite and manage members of your organization. This section details the creation of a User with the appropriate account level permissions., through to their verification and first sign on.
- Sign In for UsersThis page shows how a User logs in to Eightwire — to log in following these steps they must first have verified as a user — details are shown in the section about verification in Create Users.
Create a Project2 articles
- Project SetupA Project is a workspace for Datastores and Processes. It provides the business or use case separation within your Account. Assign Users to a Project in the capacity they need by assigning them Project Roles.
- Assign Project UsersTo assign a user to a project you must have the role of Account Administrator. Each user that is added to a project is assigned a Project role which provides them with specific Project permissions. Refer to our page about managing users for a full description of the roles and their capabilities.
Install an Agent7 articles
- Agent — Architecture for Data AccessData Access The Agent requires access to your data, otherwise it can't do its job. You should allow specific permissions to the data that you want the Agent to work with and deny access to anything you don't want the Agent to be able to access. It is good practice to only allow access to resources that you want access to and broadly deny access to everything else. You also have full control over detail about the data the Agent accesses through the web portal, please refer to Data Sharing Principles for a summary of object, column, and row level filtering.
- Agent System RequirementsMany data repositories are on internal networks behind corporate firewalls. To allow access to that data an Agent installation is required. An Agent is a small piece of software that runs inside a customer's network and provides both read and write data access capability and a secure communications channel to Eightwire's cloud services. If a Data Store uses an Agent, the connection information given should be relative to that Agent, because it is acting on Eightwire's behalf. An Agent does not usually require any modification to your firewall rules in order to operate and is very easy to install and get connected.
- Install an AgentBefore installing an Agent, make sure you have the following ready: local administrator privileges on the computer or server where the Agent is to be installed a service account that the Agent will use - created specifically or use the Default NT AUTHORITY\System. Internet access for outbound communication with the Eightwire cloud service.
- Configure an AgentThis page shows you to make changes to an installed Agent - such as changing permissions, manually updating or deleting the Agent. You must have local administrator privileges on the computer to configure the installed Agent
- Authenticate an AgentAn Agent must be authenticated by generating and registering a unique key. This is a requirement after a new Agent installation and for existing Agents after the update on 30 August 2022 to Agent version 2.2022.830.1 At any time you can generate and register a new key for your Agent. We recommend that you register a new key at least once a year. Manual and Automatic updates to the Agent do not require the registration of a new key.
- Agent FAQ's and Connectivity TroubleshootingIn the case where you have an installed Agent but it is offline — there are a number of checks you can make.
- Agent UpdatesThis page shows you how to update an Agent. When Eightwire makes a change that requires an Agent update, we will advise the change by email with at least two weeks notice. If your organisation has users that need to be part of the change distribution list, just drop us a line on support@eight-wire to be included.
Set up a Datastore5 articles
- Create and ConnectA Datastore is a view of your data repository — it defines how Eightwire will connect to the data. A Datastore can be a source or a destination and each one can vary in the required connection inputs and in its features. For example, a CSV Source datastore allows for a wildcard to be set and a datastore on a relational database allows for queries to be built in the datastore. To see all the available connectors and their features go to: Connectors
- Scan, Browse and Edit ObjectsObjects in a datastore can be scanned and browsed - however actual data will never be visible. Instead certain metadata can be browsed and edited as required - such as column names, datatypes and filters. It is very important to check and verify the data structure that is inferred by the datastore scan. The more data means the scan has a better chance of accurately profiling the structure but sometimes you may want to edit the profile by editing objects in the datastore.
- Datastore Scan FAQ'sA Datastore has options for the scan to keep it updated automatically (every 20 minutes) or manually when requested by a User. This page has information about how the scan works, so you can determine the best setting to select for your datastore
- Share your DatastoreTo allow a datastore to be visible in another Account, there is an activity to share the details and access securely. This is the 'secure handshake' that allows a datastore to used in a data transfer by another Organisation and only happens once. This page details the steps to take when sharing;
- Accept a Datastore Share InvitationIf a datastore is shared with you from another Account or Project, you will receive an email generated from the Account of the organisation sharing the datastore. This email will explain who has invited you to share data and if there are any data sovereignty restrictions. This page shows the steps required to accept the share;
Create a Process9 articles
- Process Groups and TypesA process is the instruction to transfer data. Many separate processes can be contained within a group that share a process type and may have a schedule. If the objects (tables) contained within the Processes in the group relate to each other, these relationships will be respected during data transfer when the Process Group itself is executed (e.g. with relational database referential integrity constraints).
- Create, Map, and Filter a Process
- Create Process ExpressionsProcess expressions allow you to use data attributes presented in a source datastore in expressions to build transformations that map to your destination. The syntax for the expressions and available functions are listed in detail in our section on Process Expressions Syntax.
- Use Process Constants and System ExpressionsSystem Expressions and Constants are a way to add a column of data to a process, that is independent of the source data. The column can help with audit ability of the processes, for example when you are loading data from multiple sources into a common destination.
- Process Options, Thresholds and Toleration
- Execute a ProcessAn individual process — from one source to one destination — can be executed manually, or a process group can also be run.
- Process Group Time ScheduleEightwire allows you to schedule a process execution in the future, and for the processes to repeat. The schedule is applied to a Process Group, ensuring that all processes within the group will be executed.
- Process Event Execution ScheduleThe Execution event scheduler allows you to define explicit dependencies for processes. For example you may need to ensure all your source data has been loaded before you begin any processes that transform the data. The execution sequence can be for individual processes across the Project — not confined to a Process Group.
- Monitor Process ActivityWhen Processes are executing, or have recently finished executing, you can check on their status from within the process page or from the activity page
Account and Project Alerts2 articles
- Account AlertsConfigure email notifications for events across your entire Account using Account Alerts. Make sure Agents are online and Datastores and Processes are in operation without needing to log in to Eightwire.
- Project AlertsConfigure email notifications for events within specific Projects using Project Alerts.. This allows you to be notified about connectivity of Datastores and the execution of Processes without needing to log in to Eightwire.
Datastore Tags1 articles
Datastore Features9 articles
- Datastore WildcardsIf you need to transfer data from CSV or Fixed Width files, it is likely that they are being created with a varying filename to identify the date and time of creation. In the CSV and Fixed Width Datastores there is a function to store wildcards so that the filename used in the process is constant but each of the actual files can be processed.
- Datastore QueriesIf a Source Datastore contains objects from a relational database, then a datastore query can be used to create and transform a set of data. The query is then visible in a shared datastore and can be used in a process. There are 13 Datastores that allow datastore queries — find the full list at Connector Types
- Folder DatastoresThis type of datastore enables a file of any type to be moved — for example, PDF, Word, or Image files. Most connectors extra structured data from the platforms they connect to. The Folder connector is a binary connectors, and does not make any attempt to extract data from the platforms. Instead raw binary is read and written instead. This is represented with Eightwire as structured tabular data with a binary component.
- Tile DatastoresA Tile Datastore enables a data provider to drag a file into a data sharing tile, triggering a process to a destination in your account. The Tile Datastore is a cloud connector, meaning that the source does not need an Agent to interact with it. Although it can be used by any Licence type — Tiles are specifically designed for Tile Licence Accounts — where the sharing can be quickly deployed for ad-hoc shares of data or files.
- FTP / SFTP DatastoresEightwire enables you to write and read data with external organisations who are using SFTP or FTP Servers without any change to their existing infrastructure and without them needing to have their own Eightwire Account. We have a connectors that can read and write data using CSV or Excel files located on FTP/SFTP Servers. The data can be processed between the FTP/SFTP Datastore and any other Datastore (file or relational database) We also have a Generic FTP / SFTP Datastore - specifically for copying files regardless of their extension type. The files can be moved between FTP/SFTP Datastores and Folder Datastores.
- Fixed Width DatastoresWhen working with file sources that do not have column headers — Eightwire will attempt to identify columns in the scan using the delimiter settings, and apply arbitrary column labels. This page shows how to create a Fixed Width Datastore and options to edit the datastore to apply any documented column widths and descriptions to the datastore objects.
- Teradata DatastoresThis section details requirements to connect using our Teradata Connector.
- Datastore connection to AWSThis section details requirement examples of connections to AWS using our connectors for S3 Delimited Text, and Relational Databases in an EC2 instance.
- Salesforce ConnectorConnecting to your Salesforce Account using an Eightwire datastore enables you to read and write data to standard or custom objects. The Datastore can be used after you setup access using the Salesforce Platform Tools. Once the Datastore is connected it can be shared with another Eightwire Account or used in a process for data transfer.
Connectors3 articles
- Connector TypesEach Data Store makes use of a Connector to access data on your chosen platform. Each Connector is required to perform the same basic tasks but they accomplish those tasks in quite different ways. For example, when asked to write data, an Excel Spreadsheet Connector will create an Excel file containing the data and store it in a folder system, whereas an SQL Server Connector will execute database commands to write the same data into the database. From a user perspective, both achieve the same task, but internally they work quite differently. Because all data formats have differing capabilities and limitations, so do the Eightwire connectors that work with them. Where possible Eightwire will attempt to bridge the capability gap by inferring the missing information from the data, even where this is not provided by the data format itself. For example, the Delimited Text format does not provide any way to designate a primary key column. Instead, Eightwire analyses some of your data and infers an appropriate column for you. Conversely, most database platforms explicitly allow you to specify a primary key, which Eightwire will detect and use. This page shows our current selection - there are always new connectors on our roadmap, so let us know if there is something specific you are looking for;
- Using SQL Cloud ConnectorA Datastore can connect to SQL in the cloud using the correct connection string and by allowing communication outwards to Eightwire cloud services.
- Using the Oracle ConnectorYou can connect to on-premise or cloud Oracle Datastores using the Oracle Connector.
Process Expressions Syntax10 articles
- OverviewWhen mapping Columns in a Process, instead of source data, you may choose to use a derived column. One type of derived column is an expression.
- Arithmetic Operators
- Boolean Operators
- Conditional Operators
- Precedence
- LiteralsAny expression may include literals. A literal, also known as a constant, is any non-calculated value.
- FunctionsA function takes zero or more parameters, performs some calculation, and then returns a single value. A wide range of functions are available to help you perform different tasks – these are listed in the table below. In addition, two specific functions, Data(name) and System(name) are also available, which return source table data and system metadata respectively — these are described [page here] Eightwire is generally case-insensitive. This means that you can write your expressions in uppercase, lowercase, or any combination, provided you spell things correctly. When providing function parameters for comparison/searching type activities, these will work no matter what their case, provided they are otherwise an exact match – pay attention to any spaces or other punctuation, these count when searching or comparing things.
- Control Functions for Activity LogWhere the other functions described will write data to a destination column — control flow —functions are designed to write arbitrary data to the Batch Log, with different levels of severity. In the case of the Error function, this will additionally cause the Batch to abort. These can be used to log specific conditions, monitor the data for specific conditions and generate warnings, or in extreme cases, force the batch to abort before the Write phase has begun if some condition has arisen that you wish to avoid. These functions should be used very carefully. Consider whether the tolerances in Process options can achieve the same result.
- Error Handling
- Using System DataSystem Data is the metadata surrounding an Eightwire Batch and the Process it’s based on. A Batch is the one-time execution of a Process. The Batch does the work, while the Process defines what should be done.
Videos1 articles
Associated Accounts1 articles
Build Tile Datashares6 articles
- Tile Datastore TypesA Tile is a cloud datastore, that an Account receiving data builds in their Project. It provides a way for another Account (usually Tile Licence) to manually initiate a share of the data by dragging a file into the Eightwire Tile.
- Define your Tile DatastoresThis page shows you how to create the Tile as a the party receiving the data.
- Process CSV and Excel Data using TileWhen you have a Tile Datastore set up you need to set map it in a process to a destination before sharing it to your data providers.
- Process Generic Files using TileOnce you have a Generic Tile Datastore created, you need to map it in a process to a destination folder Datastore before sharing it with your data providers.
- Share a Tile DatastoreA Tile Datastore can be shared to a full license Account or to an Associated Tile Licence Account. This is the final step in enabling a Tile Datashare.
- Monitor Tile ActivityActivity reporting for Tile processes are are available to both the source and the destination Accounts, enabling them to see the history of batches run.
Tile for Data Providers4 articles
- Tile Account: Signup & VerificationJoin Eightwire as a Data Provider and work with Tiles for manual data submission.
- Accept a Tile Datashare InvitationThe first step in being able to submit data to another Account is to accept a Datashare invitation. Once accepted - the tile that you submit data into is visible in your Account. An Associated Account will generate a Datashare invitation to your Account. The email will be received by all the users in your account.
- Submit Data using TileYour Dashboard has tiles designed to upload specific sets of data - the format of the data is determined by the Destination Account.
- Tile Account Administration
Data Preparation hints2 articles
- Excel Data ShareAre you using Excel as a source or destination? Here are some important points to consider when you are preparing data to share or when you are configuring the data that is written to Excel.
- CSV / Delimited Text Data ShareAre you using CSV as a source or destination? Here are some important points to consider when you are preparing data to share or when you need to configure the data that is written to CSV.
Architecture3 articles
- Eightwire EntitiesThis page provides you an overview of the entities in your Account, to help you plan your configuration and security model.
- Data Sharing ArrangementsCustomise your data shares by using a combination of features — permissions, filters and sharing agreements. When data shares are established between parties there are several ways to control what data is shared and the visibility of objects in a datastore. Here are the options available — they become really useful once you are sharing with more than one organisation.
- Applying Column & Row level filtersIn Eightwire there are several options to restrict the data that is available in a shared datastore without changing the source data structure.
Using APIs1 articles
Copy Account or Project1 articles
Reporting and Audit3 articles
- Project Metadata ReportThe Metadata Report displays the entire configuration of a Project — the data can be downloaded into an Excel spreadsheet or a JSON file. To access the reports, a user must have the Account Administrator role or be a Project Administrator for a specific Project.
- Account Group and CommunitiesBy Default each Account has the Aged Users Report. We have some reports which can be enabled for your Account, which include the activity of any Account that is part of your community.
- Account and Project AuditAudit reports can be run to see any configuration change within an Account and Project. They can be run by Account and Project Administrators as required.
Security Certifications and Whitepaper3 articles
- Certifications and AccreditationsThe independent certifications and audits Eightwire holds, with the date each was issued or last completed.
- Technical Whitepaper
- Security WhitepaperTo help you in evaluation, planning, or implementation - please see attached a copy of our security whitepaper for your information. For a copy of our Security Whitepaper please send an enquiry to support@eight-wire.com
Help1 articles
Articles2 articles
- Column and Smart MappingThis article details how Eightwire maps a source and destination object in a Process using column mapping and smart mapping.
- Managing Datasharing with Third PartiesThis video shows how Eightwire allows you to securely share sensitive data with third parties. If you are considering deploying us within your organization, this can answer your questions around security, privacy, and usability.
Talk to a team that knows Aotearoa’s data
We work with health, government and social sector teams every day. Tell us what you’re trying to move. We’ll show you the fastest path.
Or email us directly at support@eight-wire.com