Extending stateful URLS with node filters and Expanding/collapsing modular pipelines flag

[jitu5]

Description

Resolves #1736

In this PR the stateful URLs to include additional functionalities such as

• Node type filtering (type=task,data)
• Tag filtering (tags=companies,evaluate)
• Expanding/collapsing all modular pipelines (expandAllPipelines=true)
• Reset filter button

Below are four scenarios where the possible values for new query parameters could be:

	Empty url & New browser	Empty url & Old browser	Stateful url & New browser	Stateful url & Old browser
types	task,data (default)	From local storage	From stateful url	If available, then from stateful URL, else from local storage.
tags		From local storage	From stateful url	If available, then from stateful URL, else from local storage.
expandAllPipelines	false (default)	From local storage	From stateful url	From stateful url
pipeline_id	_default_ (default)	From local storage	From stateful url	From stateful url

reset-button.mov

Development notes

• On first load query params pipeline_id, expandAllPipelines and types will always be part of url.
• For path matching URLSearchParams used instead of matchPath.
• Shortening our existing URL query params:
  'focused_id' to fid
  'selected_id' to sid
  'selected_name' to sn
  'pipeline_id' to pid

QA notes

• All tests should pass

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added new entries to the RELEASE.md file
• Added tests to cover my changes

[rashidakanchwala]

Hi LGTM thank @jitu5 ..awesome work :D

[ravi-kumar-pilla]

Hi @jitu5 , Nice work.

I am trying out the scenario below -
Lets say someone would like to share a URL - http://127.0.0.1:4141/?pid=__default__&expandAllPipelines=false. This does not have nodeTypes (i.e., I want all types including parameters). But if I open this in incognito or new browser, the url gets appended with types=task,data. Is this desired, if so how can someone ever share a url which has all 3 nodeTypes ?

Thank you

[jitu5]

Hi @jitu5 , Nice work.

I am trying out the scenario below - Lets say someone would like to share a URL - http://127.0.0.1:4141/?pid=__default__&expandAllPipelines=false. This does not have nodeTypes (i.e., I want all types including parameters). But if I open this in incognito or new browser, the url gets appended with types=task,data. Is this desired, if so how can someone ever share a url which has all 3 nodeTypes ?

Thank you

@ravi-kumar-pilla
if you see the table above for nodeType with new browser & empty url(no nodeType params), the default ( types=task,data ) will apply. this is desired. If user wanted to to share with all nodeTypes, all should be included in urls.

[ravi-kumar-pilla]

Hi @jitu5 , Nice work.
I am trying out the scenario below - Lets say someone would like to share a URL - http://127.0.0.1:4141/?pid=__default__&expandAllPipelines=false. This does not have nodeTypes (i.e., I want all types including parameters). But if I open this in incognito or new browser, the url gets appended with types=task,data. Is this desired, if so how can someone ever share a url which has all 3 nodeTypes ?
Thank you

@ravi-kumar-pilla if you see the table above for nodeType with new browser & empty url(no nodeType params), the default ( types=task,data ) will apply. this is desired. If user wanted to to share with all nodeTypes, all should be included in urls.

okay got it. I think it is a bit of design question as well. Should we make allSelected nodeTypes == noneSelected nodeTypes ? That is how the UI behaves at the moment. What do you think ? @stephkaiser @jitu5 @rashidakanchwala

[ravi-kumar-pilla]

Hi @jitu5 , Appreciate your patience 💯

I am testing the below scenario -

1. I have a tag filter with companies selected
2. I go to incognito and paste the url - it works fine
3. I try to remove the tags in the url and refresh the page, the tag re-appears as shown in the gif
4. I tested the scenario on normal browser and it has the same issue

Please let me know if you need more information. Thank you

[jitu5]

Hi @jitu5 , Appreciate your patience 💯

I am testing the below scenario -

1. I have a tag filter with companies selected
2. I go to incognito and paste the url - it works fine
3. I try to remove the tags in the url and refresh the page, the tag re-appears as shown in the gif
4. I tested the scenario on normal browser and it has the same issue

Please let me know if you need more information. Thank you

@rashidakanchwala
if see below table

• If tags is present in url it will apply on UI
• If tags is not part of url and you have new browser(no local-storage) nothing will apply
• If tags is not part of url and and if your browser's local storage contains any tag (in your case 'companies') it will applies from your browser's local storage.

this is expected behaviour.

[stephkaiser]

Hi @jitu5 , Nice work.
I am trying out the scenario below - Lets say someone would like to share a URL - http://127.0.0.1:4141/?pid=__default__&expandAllPipelines=false. This does not have nodeTypes (i.e., I want all types including parameters). But if I open this in incognito or new browser, the url gets appended with types=task,data. Is this desired, if so how can someone ever share a url which has all 3 nodeTypes ?
Thank you

@ravi-kumar-pilla if you see the table above for nodeType with new browser & empty url(no nodeType params), the default ( types=task,data ) will apply. this is desired. If user wanted to to share with all nodeTypes, all should be included in urls.

okay got it. I think it is a bit of design question as well. Should we make allSelected nodeTypes == noneSelected nodeTypes ? That is how the UI behaves at the moment. What do you think ? @stephkaiser @jitu5 @rashidakanchwala

Thanks Ravi, do you mind rephrasing or explaining in a less-technical way? Im struggling to follow :) are we discussing if the way all/none node types are displayed in the URL is intuitive to users?

I also had a question regarding the types - what are the task and data types? From the UI i see we have Nodes, Datasets, or Parameters so i'm a bit confused.

Another question - shouldn't all pipelines be expanded by default?

[stephkaiser]

Hi @jitu5 , Appreciate your patience 💯

I am testing the below scenario -

1. I have a tag filter with companies selected
2. I go to incognito and paste the url - it works fine
3. I try to remove the tags in the url and refresh the page, the tag re-appears as shown in the gif
4. I tested the scenario on normal browser and it has the same issue

Please let me know if you need more information. Thank you

it does seem a bit strange that users wont be able to tweak the URL themselves. What happens if the user clicks on the Companies tag to remove it from the UI, will the URL update?

[ravi-kumar-pilla]

Hi @jitu5 , Appreciate your patience 💯
I am testing the below scenario -

1. I have a tag filter with companies selected
2. I go to incognito and paste the url - it works fine
3. I try to remove the tags in the url and refresh the page, the tag re-appears as shown in the gif
4. I tested the scenario on normal browser and it has the same issue

Please let me know if you need more information. Thank you

@rashidakanchwala if see below table

• If tags is present in url it will apply on UI
• If tags is not part of url and you have new browser(no local-storage) nothing will apply
• If tags is not part of url and and if your browser's local storage contains any tag (in your case 'companies') it will applies from your browser's local storage.

this is expected behaviour.

okay I understand that if there is any data in local storage, we try to get that and there is no way to check if the user removed all tags. For nodeTypes we do have some default but for tags we do not have such thing and that makes this test case confusing. Basically if there is local storage, the user can only disable a tag via UI and not via URL. May be we need to think about this approach. It is a bit confusing if I want to change the URL and it keeps appending tags.

[rashidakanchwala]

http://127.0.0.1:4141/?pid=__default__&expandAllPipelines=false

Historically, parameters in the flowchart are hidden by default based on user research indicating that most users do not need to see them. This approach has not drawn complaints, suggesting it is generally acceptable. However if we see a lot of users complaining, maybe we can consider revising our approach.
I think, when node types are unspecified, the URL should automatically revert to the default view, which includes both task and data types.

[jitu5]

Hi @jitu5 , Appreciate your patience 💯
I am testing the below scenario -

1. I have a tag filter with companies selected
2. I go to incognito and paste the url - it works fine
3. I try to remove the tags in the url and refresh the page, the tag re-appears as shown in the gif
4. I tested the scenario on normal browser and it has the same issue

Please let me know if you need more information. Thank you

    [
      
    
        ![stateful_test1](https://private-user-images.githubusercontent.com/87735534/320918972-3527cf61-8813-4208-af23-2b3b85543b87.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2Nzg4NjIsIm5iZiI6MTcxMjY3ODU2MiwicGF0aCI6Ii84NzczNTUzNC8zMjA5MTg5NzItMzUyN2NmNjEtODgxMy00MjA4LWFmMjMtMmIzYjg1NTQzYjg3LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE2MDI0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU1ZmZlZGE5YzAxMmJhNzQxMTY5YjIzYjVlOGI0NWRkNDEyZTQxOGY0Y2Q2MWIzZmM0OTNhZWQ5OGMwZWI0Y2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1EHhJ9fXU64gtnbEkcyVPvKMFtaNldbbPfpckv33jfY)
      ](https://private-user-images.githubusercontent.com/87735534/320918972-3527cf61-8813-4208-af23-2b3b85543b87.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2Nzg4NjIsIm5iZiI6MTcxMjY3ODU2MiwicGF0aCI6Ii84NzczNTUzNC8zMjA5MTg5NzItMzUyN2NmNjEtODgxMy00MjA4LWFmMjMtMmIzYjg1NTQzYjg3LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE2MDI0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU1ZmZlZGE5YzAxMmJhNzQxMTY5YjIzYjVlOGI0NWRkNDEyZTQxOGY0Y2Q2MWIzZmM0OTNhZWQ5OGMwZWI0Y2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1EHhJ9fXU64gtnbEkcyVPvKMFtaNldbbPfpckv33jfY)
    
    
      
        
          
        
        
          
          
        
      
      [
        
          
        
      ](https://private-user-images.githubusercontent.com/87735534/320918972-3527cf61-8813-4208-af23-2b3b85543b87.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2Nzg4NjIsIm5iZiI6MTcxMjY3ODU2MiwicGF0aCI6Ii84NzczNTUzNC8zMjA5MTg5NzItMzUyN2NmNjEtODgxMy00MjA4LWFmMjMtMmIzYjg1NTQzYjg3LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE2MDI0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU1ZmZlZGE5YzAxMmJhNzQxMTY5YjIzYjVlOGI0NWRkNDEyZTQxOGY0Y2Q2MWIzZmM0OTNhZWQ5OGMwZWI0Y2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1EHhJ9fXU64gtnbEkcyVPvKMFtaNldbbPfpckv33jfY)

it does seem a bit strange that users wont be able to tweak the URL themselves. What happens if the user clicks on the Companies tag to remove it from the UI, will the URL update?

@stephkaiser
User can modify URL themselves. but wherever you select the tag or nodeType it updates the url as well as it update your browser's local storage. So whenever you hit URL without tag it will check for for your browser's local storage if any previously applied tag is present if yes, it applies on UI as well as it update URL

[rashidakanchwala]

@jitu5 - I faced an issue related to tags that I got in the beginning but couldn't do it again whilst testing that case Ravi mentioned above.

• Open the browser
• Select companies tag. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=companies
• Select evaluate tag. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=companies,evaluate
• Now click on the tag group deselect. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=shuttles,train

I think it should revert it back to no tags.

Team, I propose we meet tomorrow after the standup to discuss our approach to tags. Given our current way of saving, reading to local storage and reading from the URL, some of design decisions have been made that were thought most optimal while Steph was away. Let's see what we can release soon and identify areas for future improvement.

[rashidakanchwala]

I also had a question regarding the types - what are the task and data types? From the UI i see we have Nodes, Datasets, or Parameters so i'm a bit confused.

task = node,
but from user perspective, I agree we should keep the language consistent. so we should say type = data,node,parameters.

[stephkaiser]

I think a meeting to discuss this work would be great, i wasn't involved in this so would be great to get a walkthrough and discuss tags and the default view. Also agree we should always keep language consistent, I would even suggest type = datasets,nodes,parameters.

[astrojuanlu]

Please sign me up to the meeting!

[jitu5]

@jitu5 - I faced an issue related to tags that I got in the beginning but couldn't do it again whilst testing that case Ravi mentioned above.

• Open the browser
• Select companies tag. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=companies
• Select evaluate tag. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=companies,evaluate
• Now click on the tag group deselect. Updates URL to http://localhost:4141/?types=task,data&pid=__default__&expandAllPipelines=false&tags=shuttles,train

I think it should revert it back to no tags.

@rashidakanchwala I was able to reproduce this issue with group tag filter, yesterday I fix for this and now you can test it again.

[jitu5]

I think a meeting to discuss this work would be great, i wasn't involved in this so would be great to get a walkthrough and discuss tags and the default view. Also agree we should always keep language consistent, I would even suggest type = datasets,nodes,parameters.

@stephkaiser, Now types in URL matches with UI labels type=datasets,nodes,parameters thanks.

[ravi-kumar-pilla]

Great work @jitu5 !! Looks good to me

[astrojuanlu]

I'm not qualified to comment on the code and also unfortunately don't have a lot of time to do manual QA of this in the face of #1764. Please don't wait on my review 🙏🏼 Thanks @jitu5 !