AzureRM cmdlets return object with a json-like format #1384
Comments
|
hear! hear! I am astonished at the non-PowerShellness of the new AzureRM cmdlets.
I also wonder when I will be able to create a container?
Luke
From: trondhindenes
Sent: Tuesday, December 1, 2015 7:52 PM
To: Azure/azure-powershell
Subject: [azure-powershell] AzureRM cmdlets return object with a json-like format (#1384)
When running for example Get-AzureRmVm, the following is returned:
For some reason the Azure team decided to go with a "json-like" view of returned objects, making it incredibly hard to read the output - json is simply a bad choice for being consumed by humans.
Implement proper output objects by using PowerShell formatting file (.format.ps1xml).
—
Reply to this email directly or view it on GitHub.
|
|
"json is simply a bad choice for being consumed by humans" Not to mention that nothing in the PowerShell core sets a precedent like this for displaying objects. Cheers, |
|
Honestly I think it's just in how they decided to format the output, to lead folks in the direction of templates. |
|
@jeffpatton1971 don't you agree that the output is terrible? I'm finding the spacing, characters and commas increidble hard to read. If it's too hard to implement format.psxml files, just give me regular objects. Like every other PS module on the planet does. |
|
Let me answer this way. Using the image you posted above, what you see is what appears to be json output for the extensions property. For me, I kind of like being able to see (at a glance) all the properties and values in the underlying object; it's nice. My problem with this view is that the data being displayed is most likely held in ExtensionsText property (https://gist.github.com/jeffpatton1971/29e536cd0147694796cd) this is misleading at best, an outright lie at worst. I still feel this is a silly formatting issue. from your perspective and that of someone who has been building custom powershell modules for my employers for years now, I feel like you. Seriously Microsoft? In every single blessed example of powershell on Microsoft.com and beyond this is not a thing. do not start altering the formatting like this for arbitrary cmdlets on what appear to be arbitrary modules within the overarching azure powershell library...it's stupid. |
|
Adding folks from the dev team and networking team as well @huangpf @hyonholee @DeepakRajendranMsft since this is prevalent in both Compute and Networking cmdlets. Definitely good feedback. One thing I am yet to see the alternative to a JSON format that is good at displaying properties that are n levels deep. For example how to display all the information in a StorageProfile array of data disks I'm not saying it is the best way to display information but I'm trying to find an approach that strikes a balance between users who want to see all information in one place and users who don't appreciate all the brackets, quotes. If we went with a more YAML like approach, is that acceptable? Attached is a sample |
|
Just copying this over the yammer feed. Honestly, I think just not display the text fields in the output is fine. The typical output of .net properties in powershell is really very simple. I've updated my gist (https://gist.github.com/jeffpatton1971/29e536cd0147694796cd) so you can see it. Since the beginning .net objects have appeared as their type names (ie line 57 LinuxConfiguration) and in powershell we know that we can delve into that further with $vm.osprofile.linuxconfiguration to see the properties with the object. This has been the standard that Trond Hindenes has been referring to, really I think since monad. The seemingly arbitrary change to display the .net assembly as json is jarring...to say the least. That said, I get your point, you now have easy access to view all classes and subclasses and properties...nifty..works well for looking at possibly how a template should be laid out....but again...really goes against the grain. wwjd |
|
Re-reading your statement above @singhkay and I think you're trying to solve a problem, that for those of us who use powershell day in and day out, that isn't really a problem. Was this something that was discussed only internally? Just going to re-iterate I really think this just a formatting issue, change the output format back to the traditional powershell way and be done. Then either override the .ToString() method of the objects to return json, or give param -asJson to your cmdlet. |
|
If a user wants JSON output, then they pipe the VM object into I think that the recommendation from @singhkay is reasonable. At least it doesn't have all the syntactical details included in it. It still looks a bit messy, but for the time being, I don't have any better suggestions, unless someone else chimes in with an idea. ps. I strongly recommend against adding an Cheers, |
|
@pcgeek86 Adding to this from the Yammer convo The current proposed default view is to show the below object view and show the current full detailed view using a Another question, is there a reason (other than this being the PS way) why some might prefer this view over |
|
I agree with @pcgeek86 if the underlying .net is sound, then convertto-json will allow us to see that if we want to, we can also access through the typename.class.propertyname syntax. @singhkay The useful information isn't hidden, it's there and for those of us who have been using powershell since it started, this is what we're used to and have come to expect. Again, respectfully, I think you're trying to solve a problem that isn't. |
|
@singhkay I seriously find json messy to read. In addition, what Jeff/Trevor wrote. Please look at previous unfortunate decisions the Azure PowerShell team has made in terms of diverging from the regular way of doing PowerShell and take some learning from it. |
|
I think users should become familiar with the object model of a Virtual Machine. If you display all of the content on the top-level VM object, then people will think that it's a more or less "flat" structure. Instead, if you require them to "drill down" into the VM object, using properties, then they will naturally learn the structure of the object. This isn't too different from other object models, such as the Cheers, |
|
What is the status on this issue? Could we look forward for a regular output from powershell cmdlets as against a JSON view anytime soon? ConvertTo-Json exists anyway in case we need the output to be in JSON format. |
|
Update on this, we discussed this with few other teams besides Compute and the consensus is to not adopt the object view of the Get cmdlets properties as it hides too much important information e.g. below Now on the the proposed solution. We discussed with the PowerShell team, who suggested we take a look at output of the following PS cmd which displays an object on the second level. Based on this we have two designs for you folks to consider. Note in both cases you'll still be able to access subobjects the same way you do today i.e. Proposal 1
Proposal 2
|
|
I vote for proposal 1. Cheers, |
|
Thanks for the feedback @pcgeek86! @trondhindenes @jeffpatton1971 @Vijayaraghavanl @lukeb1961 Any thoughts on which proposal you prefer? |
|
Proposal #2 is better, because the semi colons are aligned, and sub-fields at various levels are clearly indented. Proposal #1’s semi colons are scattered all around, which makes it very difficult to follow the content, and it’s unclear when it goes to next level and when it’s not, i.e. |
|
+1 for proposal #2, but as someone who lives in PoSh with these cmdlets, I'm not sure there's a big problem with the current json output. For some reason this has never bit me when manipulating objects json output or not. |
|
Not sure why you couldn't just have the output in nested Hashtables as that really is simple enough and is already widely known. Again its items like this that make me wonder about the decision points being made on the development practises of this set of Microsoft "Official" Modules An example of this in action would be the objects output from the Find-Module Cmdlet and In my Opinion this is the correct way for you to output data as it can easily be manipuated and pushed along the pipeline (if required) |
|
@kilasuit Interesting, do you have an example of the nested hashtables output from Find-Module? I'm seeing just a table |
|
If you were to run the following Find-Module AzureRM | Select-object * You'll find the AdditionalMetadata property is a Hashtable with nested Hashtables added into it. |
|
@kilasuit Thanks! I see it now. The problem I find with this is that it just displays the key in the embedded object (Created, ItemType, copyright, PackageSize...). The GET VM analogous to this wouldn't give much info in the default view. You'll have to dig into each object to find the VM properties. For an example exercise, trying to find the storage account the VM data disk is in would take many interactions with the VM object versus a view that shows all properties in the above proposals. |
|
From a PowerShell mindset though I expect to have to drill down into the objects to see the whole item and its property and with that I can't understand why you would want to nor expect us to work with this in a different manner. I suppose what I'm trying to say is from my point of view your trying to fix something that in my opinion isn't broken because you think it will be easier for other end users to understand. The problem I see with this approach is that your attempting to reinvent the already existing wheel and turn it into a square which is a reoccurring theme across the whole of these modules and in my opinion this is a very bad anti-pattern for users to get to grips with PowerShell in general and this set of Azure Modules as these could be the first entry into using PowerShell so anything they learn from here they would expect to work in other areas with other modules and this just isn't the case - which will cause frustration and people will then drop using PowerShell because of it. Again that's just my opinion but I believe this is shared with the majority of the wider community. |
From the GET VM object you'll still be able to drill down in the object and access the sub-properties. For example you'll be able to do |
|
@singhkay This is getting weird. Kay, the default behavior of nigh every other powershell cmdlet is to return the straight objects. By returning json strings we are now going against the current. While it sounds like it would make sense, you actually will create a hurdle as these users will expect other cmdlets to return similar output and they will not. I love the discussion around this, but at the end of the day, I don't know if azure powershell should be a snowflake. I think azure powershell should really fall in lock step with how the rest of the cmdlets work by default. I will re-iterate what I said earlier, by all means leave your json strings in the object, just change the output format to be the regular objects that we have all come to love, use, depend and emulate in our own code. |
It won't be returning JSON strings in the proposals above. It'll be a PowerShell object with formatting applied using the format file. I've run this by the PowerShell team who like the idea of #2 proposal above.
This is confusing to me. How does applying a different visual of the object using a format file break things? Can you share an example of your usage where you think this'll break. That will probably help me understand this better. The thing is we're discussing a changing to a compute cmdlet. Unless the other teams see a clear value in presenting outputs in more PS way ( |
|
I have to agree with @kilasuit on this one. Please look at history, and the disasterous results from every attempt the Azure team has made in coming up with something "new and easy". This is the exact line of thought that lead to the "Switch-AzureMode" fiasco. Please just give us plain PowerShell objects. Look at the object output from a VM in System Center VMM (which is also a very complex type). We want Azure VMs to behave exactly like that. We do not want "shortcuts" that are well-meant for accomodating beginners, because this would differ from pretty much all other PS modules, hurting beginners in the long term. This thread has been heavily commented by some of the top-notch folks in the PS community. Please listen to them. |
|
Honestly, I'm not terribly concerned about this anymore, since I started an external project to simplify formatting of output from the Azure PowerShell module. One of the https://github.com/pcgeek86/azure-powershell-extensions |
|
Tracking the issue here. It's already been fixed. |
|
Nice one @pcgeek86. Tho I'd still like to have correct formatting in-box without any dependencies. |
|
@pcgeek86 that's amazing, but honestly you shouldn't have to do something like that just to make the azure flavor of powershell...gee...look like powershell. @trondhindenes makes an excellent point, we appreciate the idea of helping out the new guy but I think the @singhkay is proposing is wrong. help the new guy but fixing the documentation. One of the ABSOLUTE worst things about powershell is the lack of adequate documentation. @singhkay you want to help the beginner? Bring the powershell documentation to GitHub, have guys like from the community like @pcgeek86 and @trondhindenes edit it like we do for your azure documentation. I know I actively read that and when I find problems I make changes, and I think @pcgeek86 has as well. You all figure that out, you won't have to waste cycles developing intricate and complex formatting files, you won't force us to write formatting files to fix your formatting, and you will help the new guy in more ways than you can even begin to think about. |
|
Thanks guys. I've spent so many (multiple hundreds?) of hours giving feedback that gets ignored, that I've taken it upon myself to solve problems. Better to spend the time writing things that work, instead of influencing others to make good decisions, that they ultimately don't make. I'm just shifting my time investment to something that's ultimately more positive, and that's something that @trondhindenes has done as well, with his Blue PowerShell module. |
|
@jeffpatton1971 I think we have heard this feedback loud and clear. Are you specifically talking about the help documentation? The msdn reference documentation, or conceptual documentation? Help documentation is on GitHub now, just not in a form that would encourage anyone to edit it, and this content is actually used for the reference documentation. At //BUILD, we showed an early version of tools to move the documentation to MarkDown, which should help encourage contributions. This is due to roll out once the PowerShell core team has solidified their new MarkDown format and supporting tools next Month. On cponceptual documentation, we have an overhaul underway. I would like all of this to be fixed yesterday, but you should start really seeing the effects over the late Spring and Summer. |
|
@markcowl I am talking about a number of things, but the effort here seems to be to try and simplify something for users when it's not needed. My solution to this over-simplification of powershell cmdlets is to create more extensive documentation. Specifically I talking about PowerShell documentation, when this all started with PowerShell v1 the documentation wasn't too bad. There was the regular online help documentation specific to the cmdlet, then there were a whole series of "using" documents. Let's let PowerShell do what PowerShell does best, pipe objects (not json) to the pipeline. To aid the new users to PowerShell (and Azure PowerShell specifically) let's all come up with some really good documentation. Educate the user on how to use the tools, not the other way around. PowerShell does need to work for all levels of people, but making the tool do this is NOT the answer. I know that this will come off as bitchy, but there is a whole community of PowerShell folks and I like to think that a large majority of us feel very strongly about the language. In fact I know that a great many of us have become accustom to precisely how it works, and it REALLY feels as though the Azure PowerShell group is literally choosing to go a different direction. I just got back from Redmond where the OMS group pulled in several customers and sat down with us to get our opinions on what they thought was the right direction. Did something like this happen? Did someone post in the Azure Advisors PowerShell group, hey we've decided to drop object output from Azure Powershell and instead plan to drop straight json? Perhaps I missed the memo? PowerShell is an object oriented scripting language, if I want json I will get json. Going to stop my ranting now. |
|
@markcowl let me come back to this and approach it this way. If PowerShell wasn't such an excellent tool, if many of us didn't (quite literally) use it as a source of our livelihood, then we wouldn't have this passion about it. We wouldn't care. You want to output unicorns on the pipe? meh...whatev But it's an amazing and powerful tool, it's a credit to Mr. Snover and all the programmers who work on it. And speaking for myself as a person who uses it ALL the time, I feel like it's partially mine, that I have a stake in its future, it's personal. So when we get kind of crazy and snippy, it's because we care so deeply. |
Nicely said, sir. It's because we want to work together to make it to succeed, and invest a lot of personal effort in order to make that happen. |
|
Kay have you guys thought about chatting with Bruce Payette to get his feedback? Luke Sent from my Windows Phone From: Kay Singhmailto:[email protected] @jeffpatton1971 PowerShell needs to work for a broad set of customers, experts and beginners so it's always a fine line to balance the both. Let's look at some example Q&A I want my Linux VM to be secure and only accessible through SSH. Is password authentication disabled?Current scenarioThe answer would be
Number of commands run to get this information: 1 Object type output scenarioThe answer would be
Number of commands run to get this information: 2 or 3 depending on previous familiarity with PS objects Let's ask a combination of questions this time I want my Linux VM to be secure and only accessible through SSH. Is password authentication disabled? What storage account is my osdisk in? How many NICs are on my VM? What image was used for the VM?Current scenarioThe answer would be
Number of commands run to get this information: 1 Object type output scenarioThe answer would be
Number of commands run to get this information: 5 NOTE: All the above commands work in the upcoming release. Then the question for us is why would we want to increase the time it takes for the user to answer a question by having to run more commands? As an expert PowerShell user if all the objects and properties can be accessed in a PS way with a You are receiving this because you were mentioned. |
|
@singhkay just an additional comment on the examples you've provided above: There's an object attribute called "ProvisioningState", however, the format-table output shows this as "Provisioning State". This is exactly what we're talking about: Design decisions that are not thought thru. I'm pretty sure everyone in this thread agrees with me when I say that formatters that rename the "true" attribute name of an object are pretty much the worst, and this is a pretty clear sign the redesigned cmdlets have not been subject to proper end-user testing (Specifically, If I see at ft column name called "Provisioning State" I would expect to be able to do |
In most cases, yes, I agree with that. There are some rare cases where renaming a property in the formatted output is appropriate, but it should be thought through prior to making a decision. |
|
@lukeb1961 This was checked with the PS team but I'll check with Bruce as well although it looks like he's been OOF for a while. @kilasuit @trondhindenes @jeffpatton1971 It'd be good for the team to see an example of the Get-AzureRmVm output you are proposing. I think that might make things more clearer of what the community expectations are around the amount of information returned. |
|
@singhkay - I've given my example above numerous times so I'm struggling with this
Seriously I would suggest having a full re-read of this thread and having a look at the suggestions given prior including these ones that clearly point out what you guys need to be doing as these are a few of the points that others have made
So the format that I expect as default output would be Id :/subscriptions/XXX/resourceGroups/ws2016/providers/Microsoft.Compute/virtualMachines/ws2016
ProvisioningState : Succeeded
Name : ws2016
Location : westus
Type : Microsoft.Compute/virtualMachines
HardwareProfile : {Size}
StorageProfile : {Imagereference, OSDisk}
OSProfile : {ComputerName, UserName, WindowsConfiguration}
NetworkProfile : {NetworkInterfaces}As previously noted 2 months ago |
|
@kilasuit Thanks Ryan @trondhindenes @jeffpatton1971 @lukeb1961 @pcgeek86 Are you agreed on the above format? |
|
Personally, I think striking a balance between displaying the most useful information by default and leaving other information in the default format makes sense, but it seems like that isn't the consensus view on this thread. |
|
@markcowl I think all we're trying to say is let PowerShell be PowerShell, it outputs objects. I really like the JSON and I get why it's there, why not allow the object output from get-azurermvm be passed into convertto-json so that it outputs the proper json azure needs. But most of us who have been griping in this thread have been using powershell since before it was called powershell. We see and use the cmdlets that all the other groups from inside Microsoft publish and the azure group just seems like they collectively said, " I see what you did there, we're not doing that." The general gist of this discussion has been around making the powershell more accessible to folks who are new to it. I get that, I think that's important, but I think making an entire collection of cmdlets that output data in a totally different way than any other cmdlet is not right. Look at it this way, we have a new administrator and we introduce them to powershell by having them open up powershell, running a few cmdlets so they can see how things work. Then we have them load up some of the admin modules say for active directory, exchange, and walk them through some basic admin type things. As they work through they begin to realize that these blocks of text are actually objects, and many objects have properties and methods that can be called directly from them. Then we say, here is azure, the first time they run get-azurermvm ( I pick on that one for some reason ) they get an ENTIRELY different output. the learning curve just took a steep angle up, for no good reason. I said early on, I love the json output, but give me a switch param -asjson or some such, that way I can see the json when I want to see/use it. One of the guys in here went out and just wrote his own set of cmdlets/formatters to not have to deal with this. It all just seems...odd. You all have my contact info, i'm happy to hop on a call or something, i'm not trying to be a jerk, but we are passionate about how things are, and sometimes that spills out. |
|
@singhkay I agree with the proposed format from @kilasuit (which is basically no format at all) |
|
Is there any further update on this @singhkay as its now been 6 weeks since the last comment was made on this thread |
|
@kilasuit Sorry, been busy with non-PS priorities. Here's the design being currently discussed with the PS team. This is based on existing Get-Date cmdlet that has a parameter The proposal is to add a |
|
I'm not opposed to a displayhint parameter, but your reply seems to indicate that the current json output would be the default. I don't understand why you would believe that to be the preferred output for most users - I seriously believe it's the other way around. Maybe it would be a good idea to do a survey on the Azure insiders group / powershell slack group to get a broader group's opinion on this. |
|
@trondhindenes The discussion is not closed on the default view but IMO we have an opportunity to experiment here a bit on what should be the default provided the return object is not changed so that we don't have breaking changes to scripts. Here's an example experiment that could be done to further this discussion:
Thoughts?? |
|
My first thougt is that that would make an extremely long-running issue even more long-running since we're not actually fixing the problem. Again I don't see the problem: Thousands if not millions of Powershell cmdlets have already set precedence for how Powershell should work in terms of output. All you guys need to do is align to that precedence. |
|
@singhkays - Now I'm working more and more with Azure this format is really starting to irritate me. This isnt a great UX and from a design pov it just screams out that it is just a cluttered mess of output which is 95% of the time unnecessary to output all of that information. So please I ask of you to please get this sorted out! |
|
@kilasuit We have had limited dev resources to focus on the format but we're working on getting feedback from a larger community on the proposals outlined in this thread in the meantime. Here's a survey that's been sent out to different communities with more than 1000+ members (PS experts and non-experts) https://www.surveymonkey.com/r/HF5X9R8 Additionally we've been experimenting with the display format for the Get-AzureRmVm cmdlet (without parameters). Here's a preview We explored showing the current running status of the VM in this view but that means that we have to call the Instance View API on each of the VMs returned in the subscription which increased the execution time on the cmdlet and could also cause customers with large number of VMs in the subscription to run into hourly API throttling limits. |
|
We've released these improvements in PS release 3.3.0. Now https://github.com/Azure/azure-powershell/releases/tag/v3.3.0-December2016 |
bsiegel
added
the
Service Attention
When running for example Get-AzureRmVm, the following is returned:
For some reason the Azure team decided to go with a "json-like" view of returned objects, making it incredibly hard to read the output - json is simply a bad choice for being consumed by humans.
Implement proper output objects by using PowerShell formatting file (.format.ps1xml).
The text was updated successfully, but these errors were encountered: